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GLOSSARY  OF  TERMS 

AAE  -  average  absolute  error  -  A  summary  statistic  descriptive  of  the 
error  amplitude  distribution  of  a  sample  of  tracking  performance. 
Specifically,  it  is  the  mean  of  the  amplitude  distribution 
replotted  with  all  error  amplitudes  positive.  It  is  an  indicator 
of  variability. 

AE  -  average  error  -  A  summary  statistic  descriptive  of  the  error 
amplitude  distribution  of  a  sample  of  tracking  performance. 
Specifically,  it  is  a  numerical  index  of  the  central  tendency  of  the 
amplitude  distribution. 

CCIP  AUTO  FLIGHT  SEGMENT  -  CCIP  -  (Continuously  Computed  Impact  Point) 
AUTO  is  an  A-7  bomb  mode  used  to  attack  visually  acquired  targets 
using  dive  or  low  altitude  level  delivery. 

COLOR-CODING  CONDITION  -  Experimental  condition  in  which  the  symbols 
were  both  shape-  and  color-coded  to  denote  their  states.  Red  was 
used  to  denote  hostile,  yellow  unknown,  and  green  friendly. 

COMPARING  QUESTIONS  -  Questions  in  which  the  pilots  were  required  to 
compare  specified  symbols  in  two  quadrants  of  the  format. 

COMPLEX  QUESTIONS  -  Questions  in  which  pilots  were  required  to 

recognize  symbol  orientation  as  well  as  symbol  type  and  state. 

Half  of  each  type  of  question  (Counting,  Searching,  Comparing)  were 
complex.  Complex  questions  involved  only  aircraft  symbols,  not  SAM 
or  AAA  symbols. 

COMPOSITE  SYMBOL  -  The  combination  of  symbol  type  (*,  S,  or  A)  and 
symbol  designator  (/\,  m  ,  or^)  to  form  one  symbol. 

COUNTING  QUESTIONS  -  Questions  in  which  pilots  were  asked  the  number 
of  a  specified  symbol  on  the  format. 
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GLOSSARY  OF  TERMS  (Cont'd) 

CRUISE  FLIGHT  SEGMENT  -  CRUISE  is  an  A-7  flight  mode  used  between  CLIMB 
and  DESCENT. 

HORIZONTAL  SITUATION  FORMAT  (HSF)  -  The  format  located  on  the  upper 
center  CRT.  The  HSF  presented  the  combat  situation  symbology  and 
some  navigation  information. 

KOLMOGOROV  SMIRNOV  TEST  -  A  one-sample  goodness  of  fit  test  to 

determine  if  the  distribution  of  a  set  of  sample  values  differs 
from  the  normal  distribution.  Used  to  analyze  questionnaire 
rating-scale  data. 

KRISHNAIAH  FINITE  INTERSECTION  TESTS  (FITs)  -  A  set  of  tests  conducted 
after  significant  MANOVA  results  are  found  to  determine:  1)  which 
of  the  dependent  variables  were  most  sensitive  to  changes  in 
independent  variables;  and  2)  which  of  the  experimental  groups 
differed  significantly  from  each  other. 

MULTIVARIATE  ANALYSIS  OF  VARIANCE  (MANOVA)  -  A  statistical  procedure 

which  takes  into  account  the  fact  that  several  partially  correlated 
dependent  variables  may  be  affected  by  experimental  manipulation, 
and  which  can  determine  significant  differences  in  experimental 
conditions. 

RASTER-WRITTEN  CRT  -  A  CRT  in  which  the  picture  is  generated  from  the 
top  left  corner  to  the  lower  right  corner  by  a  series  of  horizontal 
actions. 

ROOT-MEAN-SQUARE  ERROR  (RMS)  -  A  summary  statistic  descriptive  of  the 
error  amplitude  distribution  of  a  sample  of  tracking  performance. 
Specifically,  it  is  an  index  of  performance  variability  that  is 
relative  to  the  null  point. 
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GLOSSARY  OF  TERMS  (Cont'd) 

SEARCHING  QUESTIONS  -  Questions  in  which  the  pilots  were  required  to 
search  for  a  specified  symbol  across  all  quadrants  of  the  format. 

SHAPE-CODING  CONDITION  -  Experimental  condition  in  which  the  symbols  were 
shape-coded  only  to  denote  their  states. 

SIMPLE  QUESTIONS  -  Questions  in  which  pilots  were  required  to  recognize 
only  symbol  type  and  state.  Half  of  each  type  of  question 
(Counting,  Searching,  Comparing)  were  simple. 

STANDARD  DEVIATION  (SD)  -  A  summary  statistic  descriptive  of  the  error 
amplitude  distribution  of  a  sample  of  tracking  performance. 
Specifically,  it  reflects  the  variability  of  dispersion  of  the 
meansures  around  the  central  tendency  (as  indexed  by  the  absolute 
error  or  AE). 

STROKE- WRITTEN  CRT  -  A  CRT  on  which  vector  graphic  figures  are  drawn  by 
an  electron  beam  much  like  they  would  be  by  a  pen  or  pencil. 

SYMBOL  DENSITY  LEVEL  -  Independent  experimental  variable  in  which  one- 
third  of  the  HSF  formats  contained  10  symbols,  one-third  20 
symbols,  and  one-third  30  symbols. 

SYMBOL  STATE  -  Symbols  could  be  in  any  of  three  states:  hostile, 

unknown,  or  firendly.  In  the  shape-coded  condition  hostile  symbols 
were  denoted  by  a  A  hat,  unknown  a  i — i  hat,  and  friendly  an  hat. 

In  the  color-coded  condition  hostile  symbols  were  also  red,  unknown 
yellow,  and  green  friendly. 

SYMBOL  TYPE  -  There  were  three  symbol  types:  anti-aircraft  artillery 
(AAAs),  surface-to-air  missiles  (SAMs),  and  aircraft.  AAA  were 
denoted  by  the  sumbol  A,  SAMs  by  the  symbol  S,  and  aircraft  by  a*. 
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GLOSSARY  OF  TERMS  (Concluded) 

WILCOXIN  MATCHED-PAIRS  SIGNED  RANKS  TEST  -  A  statistical  test  which 
utilizes  information  about  the  direction  of  the  differences  among 
pairs  to  determine  significance. 
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SUMMARY 


an 


The  Department  of  Defense  is  currently  working  on  a  real-time 
combat  situation  display  to  be  used  in  fighter  aircraft  cockpits.^ This 
study  evaluated  a  symbology  set  for  this  type  of  display  under  simulated 
combat  mission  conditions.  The  primary  purpose  of  the  study  was  to 
compare  pilot  flight  performance  and  symbol  recognition  performance  using 
two  different  coding  conditions:  (a)  one  in  which  symbols  were  shape- 
coded  only;  and  (b)  one  in  which  symbols  were  both  shape-  and  color- 
coded.  These  two  coding  conditions  were  tested  under  three  different 
symbol  density  levels:  10,  20,  and  30  symbols.  The  addition  of 
redundant  color-coding  to  shape-coded  symbols  was  found  to  reduce  the 
overall  average  response  time  to  symbology  questions  by  forty  percent. 

The  effectiveness  of  redundant  color-coding  became  more  pronounced  as 
the  symbol  density  of  the  situation  formats  increased. 


A  secondary  purpose  of  the  study  was  to  compare  performance 
differences  among  the  three  symbol  types  (A  for  anti-aircraft,  S  for 
surface-to-air  missile,  and  *  for  aircraft)  in  each  of  the  three  symbol 
states  ('\for  hostile,  i — i  for  unknown,  and  ^  for  friendly).  The 
analyses  found  the  A  and  'S'  symbols  required  significantly  longer 
times  for  identification  than  any  other  symbols.  This  is  thought  to  be 
due  to  the  state  designators  being  parallel  to  the  top  of  the  letter  in 
both  cases.  Subjectively  the  pilots  disliked  the  A  symbol  because: 

(1)  it  closely  resembled  a  delta-shaped  symbol  (*^)  that  in  the  past  has 
been  used  to  denote  aircraft;  and  (2)  they  associate  the  letter  A  with 
aircraft  rather  than  anti-aircraft  artillery.  The  subjective  data  also 
revealed  the  pilots  had  some  difficulty  distinguishing  ther">  and  A 
state  designators  due  to  the  characteristics  of  the  raster  display. 
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SUMMARY  (Cont'd) 

Performance  differences  as  a  function  of  question  type  and 
complexity  were  also  examined.*  The  counting  questions  had 
significantly  longer  response  times  and  a  significantly  higher  error 
rate.  The  longer  response  time  for  counting  questions  is  accounted  for 
by  the  fact  that  the  pilot  had  to  do  an  exhaustive  scan  of  the  whole 
display  for  this  type  of  question,  whereas  he  did  not  for  searching  or 
comparing  questions.  This  longer  response  time  supports  the  use  of 
subscripts  to  denote  clusters  of  a  particular  symbol  on  a  situation 
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SECTION  I 
INTRODUCTION 


1.  BACKGROUND 

The  feasibility  of  implementing  a  real-time  combat  situation 
display  in  a  fighter  aircraft  is  currently  being  pursued  by  the 
Department  of  Defense  (DOD).  Such  displays  would  have  vital  threat 
location  information  transmitted  to  them  via  a  battle  area  communications 
network  consisting  of  satellites  over,  or  aircraft  and  ships  near,  the 
forward  edge  of  the  battle  area.  These  communications  networks  would,  of 
course,  be  secure  and  jam  resistant.  The  transmissions  would  be  matched 
with  the  inertial  navigation  system  of  each  fighter  aircraft,  and  the 
resultant  combat  situation  display  would  locate  the  threats  in  relation 
to  that  aircraft's  position. 

One  crucial  step  In  the  development  of  such  a  real-time  situation 
display  Is  to  devise  a  symbology  set  that  can  be  used  both  with  newer 
aircraft  coming  into  the  DOD  fighter  Inventory  having  head-down  raster- 
written  cathode  ray  tubes  (CRTs)  such  as  the  F-16  and  A-10  and  with  the 
current  F-14  and  F-15  aircraft  which  have  a  combination  of  stroke-  and 
raster-written  CRTs. 

The  Flight  Dynamics  Laboratory  Implemented  one  such  symbology  set 
on  a  simulator  cockpit  CRT  having  raster  writing  (see  Table  1  for 
symbology  set).  Each  symbol  conveyed  three  important  details:  1) 
whether  it  was  an  aircraft  (•),  surface-to-air  missile  (s),  or  anti¬ 
aircraft  artillery  (A);  2)  whether  it  was  hostile  (A),  unknown  (< — i),  or 

friendly  (O);  and  3)  in  the  case  of  an  aircraft,  Its  heading  (12,  3,  6, 
or  9  o'clock). 

A  factor  of  primary  concern  was  whether  the  pilot  of  a  fighter 
aircraft  could  effectively  use  such  a  display  during  the  heavy  workload 
situations  found  In  combat.  The  questions  were:  1)  would  the  display 
prove  to  be  too  complicated  to  be  Interpreted  by  the  pilot;  and  2)  would 
it  be  too  much  of  a  distraction  from  his  primary  flying  and  weapon 
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TABLE  1 


SYMBOLOGY  SET  USED  IN  THE  EVALUATION 


Height  Width  Type  State  Orientation 

Symbol  (mm)  (mm) _ 


i 

3 

6 

3 

6 

6 

3 

6 

3 

*  (Aircraft) 

xv  (Hostile) 

12  o’clock 

3  o’clock 

6  o'clock 

9  o’clock 

m 

3 

6 

12  o'clock 

3 

6 

3 

*  (Aircraft) 

n  (Unknown) 

3  o’clock 

LU 

3 

6 

6  o’clock 

E 

6 

3 

9  o’clock 

r\ 

3 

6 

12  o’clock 

0 

6 

3 

•  (Aircraft) 

r\  (Friendly) 

3  o’clock 

3 

6 

6  o’clock 

E 

6 

3 

9  o'clock 

f 

n 

6 

(Hostile) 

's’ 

■S 

S  (Surface-to-air  missile) 

r-i  (Unknown) 

12  o’clock 

mm 

19 

H 

(Friendly) 

H 

6 

m 

/v  (Hostile) 

mm 

6 

H 

A  (Anti-aircraft  artillery) 

n  (Unknown) 

12  o'clock 

B 

6 

r \  (Friendly) 
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delivery  tasks.  This  study  provides  both  objective  performance  data  from 
pilots  using  the  display  under  simulated  mission  conditions,  and  pilot 
opinions  on  the  usefulness  of  such  a  display  under  combat  conditions. 
Pilots  were  given  recognition  tasks  involving  situation  formats  with 
density  levels  of  10,  20,  and  30  symbols  to  simulate  varying  degrees  of 
activity  in  the  combat  area. 

Previous  research  in  the  area  of  displays  has  shown  redundant 
color-coding  (e.g.  coding  symbols  both  by  shape  and  color)  to 
significantly  reduce  both  symbol  recognition  time  and  number  of  errors 
(References  1,  2,  and  3).  Therefore,  one  experimental  condition  of  this 
study  presented  situation  formats  whose  symbols  were  shape-coded 
according  to  state  (A  for  hostile,'  i  for  unknown,  and  ^  for 
friendly),  while  a  second  condition  also  had  each  of  the  symbols 
additionally  color-coded  (red  for  hostile,  yellow  for  unknown,  and  green 
for  friendly). 

However,  despite  this  research  supporting  the  use  of  color  as  a 
beneficial  coding  technique,  not  all  research  in  the  area  has  been 
favorable.  Several  authors  have  stressed  that  the  impact  of  color-coding 
Is  highly  situation-specific  and  may  have  neutral  or  even  detrimental 
effects  in  given  situations;  the  effect  depends  on  a  number  of  diverse 
factors  such  as  operator  task,  display  medium,  or  display  environment 
(References  4  and  5).  It  has  also  been  pointed  out  that  most  experiments 
investigating  color-coding  have  used  subjects  who  devoted  their  full 
attention  to  the  display  and  performed  single,  relatively  simple  tasks 
(Reference  4). 

2.  PURPOSE 

This  experiment  did  not  limit  the  function  of  the  operator  to  the 
performance  of  one  simple  task  at  a  time,  nor  did  it  dictate  an  optimal 
display  environment  or  display  medium.  The  study  was  designed  to 
evaluate  redundant  color-coding  for  a  real-time  situation  format  in  a 
fighter  aircraft  cockpit.  Such  a  format  would  be  used  In  somewhat 
peripheral  manner  by  a  highly-loaded  pilot  while  performing  complex 
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flight  maneuvers.  Thus,  this  study  sheds  some  light  on  the  effectiveness 
of  redundant  color-coding  for  formats  used  by  operators  who  are 
performing  a  series  of  complex  tasks  which  allow  them  little  time  to  scan 
the  display. 

Therefore,  the  primary  purpose  of  this  study  was  to  compare  pilot 
flight  performance  and  symbol  recognition  performance  under  simulated 
flight  conditions  using  two  different  coding  conditions:  (a)  one  in 
which  symbols  were  shape-coded  only;  and  (b)  one  in  which  symbols  were 
both  shape-  and  color-coded.  These  two  coding  conditions  were  tested 
under  three  different  symbol  density  levels:  10,  20,  and  30  symbols. 

A  second  factor  examined  in  the  study  was  the  distinguishability  of 
the  state  designators  (A,i  l ,  and  ^v)  and  the  symbol  types  themselves 
(•,  S,  and  A).  The  state  designators  were  of  particular  concern, 
because  as  the  aircraft  symbols  traveled  there  were  distortions  of  the 
raster -written  state  designators— particularly  in  the  3  and  9  o'clock 
orientations. 


Figure  1.  Fixed-Base,  Single-Seat  Cockpit  Simulator 
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a.  Electro-optical  Displays 

For  this  study  the  head-up  display  (HUD)  and  three  of  the  five 
head-down  displays  were  active  (see  Figure  2).  All  active  displays  were 
raster  CRTs,  and  were  named  according  to  the  type  of  formats  which  were 
presented  on  them  for  this  particular  study.  The  primary  flight 
symbology  appeared  directly  in  the  pilot's  forward  field-of-view  on  the 
HUD  (see  Figure  3).  The  horizontal  situation  format  (HSF)  was  used  to 
present  the  combat  situation  symbology  and  some  navigation  information. 
For  use  during  the  experiment,  the  face  of  the  HSF  was  marked  off  into 
four  quadrants,  with  the  quadrant  numbers  clearly  marked  on  the  HSF  bezel 
(see  Figure  4).  The  order  used  in  numbering  the  quadrants  was 
determined  by  surveying  a  number  of  pilots,  and  implementing  the  order 
preferred  by  the  majority.  Relevant  navigation,  communications,  and 
stores  information  appeared  on  the  status  format  (SF)  (see  Figure  5). 

The  multifunction  keyboard  (MFK)  was  used  to  present  the  current  legends 
of  the  five  multifunction  switches  located  down  each  side  of  the  display 
(see  Figure  6).  Further  discussion  of  the  MFK  is  deferred  to  subsection 
lb  of  Section  II.  Comprehensive  narratives  of  all  four  of  the  displays 
can  be  found  in  Appendix  A. 

b.  Combat  Situation  Formats  for  the  HSF 

A  total  of  27  combat  situation  formats  were  generated  for  the 
experiment  (see  Appendix  B  for  examples  of  formats).  Three  of  these 
formats  were  used  for  the  cockpit  briefings,  six  for  the  training 
missions,  and  eighteen  for  the  data  missions.  Within  each  of  these  three 
functional  categories,  one-third  of  the  formats  contained  10  symbols, 
one-third  20  symbols,  and  one- third  30  symbols. 

Each  format,  regardless  of  density  level,  contained  symbols  for 
aircraft  (•  ),  SAMs  (S),  and  AAAs  (A).  Each  of  the  above  symbols  was 
augmented  by  a  state  designator:  hostile  (A),  unknown  (i — i),  or  friendly 
(<^)  (see  Appendix  B).  In  the  color-coded  condition,  the  hostile  was 
also  red,  the  unknown  yellow,  and  the  friendly  green.  A  ratio  of  1  S: 

1  A:3 •  and  a  ratio  of  5  A  :  3<— >:  lr\  was  maintained  for  each  of 

the  27  formats.  These  ratios  were  chosen  based  on  the  perceived  threat 
mix  that  a  pilot  may  encounter  in  a  tactical  environment. 


Location  of  Electro-Optical  Display  Formats 
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Figure  4.  Horizontal  Situation  Format  (HSF) 
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The  symbol  for  SAMs  (S)  and  AAAs  (A)  remained  stationary  in  a  fixed 
position  relative  to  the  inertial  track  and  were  presented  in  only  one 
orientation— pointed  toward  12  o'clock.  The  aircraft  symbols  (•) 
appeared  in  any  of  four  orientations  (12,  3,  6,  or  9  o'clock)  and 
traveled  in  that  direction.  Each  aircraft  orientation  appeared  an  equal 
number  of  times  across  all  formats.  Because  the  pilot's  aircraft  symbol 
traveled  along  the  inertial  track,  the  combat  situation  format  appeared 
to  move  toward  the  bottom  of  the  display  or  the  6  o'clock  position  as  the 
pilot  flew  along  the  groundtrack. 

Each  combat  situation  format  represented  a  combat  environment  80 
nautical  miles  in  length.  The  pilot's  aircraft  symbol  was  located  at  the 
center  of  the  environment  with  a  range  of  40  nautical  miles  ahead  of  it. 
Symbol  type,  symbol  state,  aircraft  symbol  orientation,  and  the 
horizontal  and  vertical  location  of  each  symbol  on  the  format  were 
randomly  assigned  within  the  constraints  of  the  experimental  design  so 
that  the  proper  ratios  among  symbol  types,  symbol  states,  and  aircraft 
orientations  were  maintained. 

The  formats  were  computer- generated,  and  were  overridden  by  the 
groundtrack.  The  symbols  on  each  format  were  prioritized  according  to 
the  degree  of  threat  they  posed  to  the  pilot's  aircraft,  and  were 
programmed  such  that  when  two  symbols  crossed,  the  more  lethal  threat 
always  overrode  the  lesser  one.  The  descending  order  of  lethality  used 
was:  hostile  SAMs,  hostile  AAAs,  hostile  aircraft,  unknown  SAMs,  unknown 
AAAs,  unknown  aircraft,  friendly  SAMs,  friendly  AAAs,  friendly  aircraft. 
Each  symbol  retained  the  same  state  of  lethality  throughout  the 
missions.  Symbols  of  interest,  that  is,  symbols  about  which  questions 
were  to  be  asked,  were  checked  out  to  Insure  that:  1)  they  did  not  run 
into  another  symbol (s);  2)  they  did  not  disappear  off  the  display;  and 

3)  for  comparison  type  questions,  they  did  not  cross  from  one  quadrant 
of  the  display  into  another. 
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c.  Multifunction  Keyboard  (MFK)  and  Data  Entry  Keyboard 
(DEK)  Hardware 

(1)  MFK  Hardware 

The  MFK  hardware  consisted  of  eight  dedicated  push-button 
function  select  switches  in  a  row  across  the  top  of  the  left  CRT,  and 
five  push-button  multifunction  switches  down  each  side  of  the  CRT  (see 
Figure  6).  Seven  of  the  eight  function  select  switches  were  lighted 
during  the  study,  and  their  respective  legends  displayed  on  the  switch 
faces.  The  MFK  was  programmed  according  to  a  tailored  logic  based  on 
frequency  of  use.  The  ten  multifunction  switches  controlled  the  ten 
functions  the  pilot  was  most  likely  to  require  in  the  particular  flight 
segment  he  was  in.  The  multifunction  switches  were  operable  to  the  pilot 
only  after  the  experimenter  initiated  a  task  event.  They  remained 
operable  until  the  task  event  was  completed.  The  function  select 
switches  across  the  top  of  the  CRT  were  rendered  inoperable  to  the  pilot 
because  during  the  experiment  the  pilot  was  asked  only  to  perform  MFK 
tasks  which  could  be  initiated  directly  from  one  of  the  ten  active  MFK 
switches. 


(2)  DEK  Hardware 

A  Data  Entry  Keyboard  (DEK)  consisting  of  twelve 
dedicated  push-button  keys  arranged  in  a  4  x  3  telephone  layout  was 
located  on  the  forward  section  of  the  left  side  console  (see  Figure  7). 
The  CLEAR  and  ENTER  keys  on  the  DEK  appeared  on  the  left  and  right  sides 
of  the  zero,  respectively.  In  addition,  the  letters  N,  S,  E,  W,  X,  and 
Y  could  be  selected  on  the  keys  labeled  2,  8,  4,  6,  7,  and  9 
respectively.  (The  computer  could  discern  whether  letter  or  number  data 
entry  was  required,  and  therefore  no  shifting  device  was  necessary.)  The 
DEK  became  operable  and  lighted  when  the  task  required  the  pilot  to 
enter  digits,  and  remained  in  this  condition  until  the  ENTER  key  had  been 
pushed  and  the  data  accepted  by  the  computer. 
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Figure  7.  Data  Entry  Keyboard  (DEK) 


d.  Dedicated  Controls  and  Displays 

Most  of  the  backup  flight  instruments  in  the  cockpit  simulator 
were  inoperable  so  that  the  pilot  was  forced  to  use  the  information 
displayed  on  the  HUD  and  HSF  to  maintain  control  of  the  simulator. 
However,  the  following  devices  were  operable  and  available  for  use  by  the 
pilot  during  the  experiment: 

a)  Angle  of  Attack  Indicator 

b)  Engine  Instrumentation  (RPM,  Turbine  Outlet  Pressure, 

Fuel  Flow  Indicator,  Turbine  Outlet  Temperature,  Oil  Pressure) 

c)  HUD  Declutter  Switch 

d)  Master  Arm  Switch 

e)  Control  Stick  and  Rudders 

f)  Trim  Button  and  Bomb  Release  Button  on  the  Stick 

g)  Throttle 
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At  the  onset  of  the  experiment,  the  cockpit  displays  were  tailored 
to  the  cruise  mode  and  the  CRUISE  flight  segment  switch  located  below 
the  HUD  was  lighted.  Appendix  A  provides  a  more  complete  description  of 
the  dedicated  instruments  and  switches. 

2.  EXPERIMENTERS'  CONSOLE 

The  experimenters'  console  provided  repeater  displays  which  allowed 
the  experimenters  to  monitor  the  cockpit  displays  and  provided  status 
lights  which  allowed  them  to  monitor  the  pilot's  performance  on  MFK  tasks 
(see  Figure  8).  A  series  of  console  push-button  switches  allowed  the 
experimenters  to  control  task  initiation,  task  termination,  data  flight 
termination,  and  recording  of  task  response  times.  The  experimenters' 
console  also  contained  a  status  display  not  found  in  the  cockpit  (see 
Figure  9).  This  display  presented  pertinent  experimental  details  such  as 
task  type,  mission  task  number,  time  into  task,  and  elapsed  time  since 
mission  start.  Also  presented  were  the  digits  programmed  for  the  MFK 
tasks,  the  digits  selected  by  the  pilot,  and  the  correct  responses  for 
the  symbology  questions.  A  Sony  cassette  tape  recording  device  housed 
in  the  experimenters'  console  permitted  the  recording  of  voice  trans¬ 
missions. 

3.  PDP  11/50  KEYBOARD  TERMINAL 

The  training  and  data  missions  were  Initialized  through  the  PDP 
11/50  terminal.  The  pilot's  responses  to  the  symbology  questions  were 
also  entered  through  the  terminal  by  one  of  three  experimenters  during 
the  experiment. 


4.  ADDITIONAL  FACILITY  SYSTEM  ELEMENTS 

A  functional  description  of  each  system  element  of  the  simulation 
facilities  is  given  in  Appendix  C.  This  appendix  also  provides  a  layout 
and  description  of  each  piece  of  equipment  on  the  experimenters' 
console. 
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Figure  8.  Experimenters'  Console 


CURRENT  TASK  1 

ELASPED  TIME  3:10 

TASK  TYPE  8 

MSN  T1 

SEQ  • 

SEGMENT  2 

MATRIX  2 

GROUND  SPEED  430 

PILOT  1 

PITCH  DEVIATION -178 

CONFIGURATION  1 

BANK  DEVIATION'**  8 

Figure  9.  Experimenters'  Console  Status  Display 
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SECTION  III 
TEST  METHOD/APPROACH 

1.  TEST  OBJECTIVE 

The  primary  purpose  of  this  study  was  to  compare  pilot  flight 
performance  and  symbol  recognition  performance  under  simulated  flight 
conditions  using  two  different  coding  conditions.  Under  one  condition 
each  of  the  situation  formats  had  shape-coded  symbols,  while  under  the 
other  condition  the  same  formats  had  redundant  color-coding  added  to  the 
shape-coding. 

A  secondary  purpose  was  to  determine  If  performance  differences 
occurred  between  the  two  coding  conditions  as  a  result  of  Increasing 
situation  format  symbol  density.  To  test  this  hypothesis,  each  of  the 
two  coding  conditions  was  tested  under  three  symbol  density  level 
conditions:  10,  20,  and  30  symbols. 

A  third  purpose  was  to  examine  the  dlstlngulshablllty  of  the 
proposed  symbol  set's  state  designators  and  type  designators  when  used  In 
combination.  There  were  nine  symbol  combinations:  'ft,  'ft,  'ft  .HT, 

'ft,  'ft,  and ^  . 

Five  objective  performance  measures  were  used  In  this  study.  Three 
pertained  to  pilot  flight  performance— true  airspeed  deviation  from  350 
knots,  and  vertical  and  horizontal  deviations  from  the  commanded  flight  path; 
and  two  pertained  to  symbol  recognition  performance— response  time  and 
accuracy.  Subjective  measures  were  obtained  through  the  use  of  three 
debriefing  questionnaires. 


2.  TEST  DESIGN 

Each  pilot  flew  four  thirty-minute  data  missions:  two  with 
achromatic  shape-coded  situation  format  symbology,  and  two  with  color- 
and  shape-coded  situation  format  symbology.  Half  of  the  pilots  flew 
missions  with  achromatic  coding  first,  half  flew  missions  with  color¬ 
coding  first— pilots  having  been  randomly  assigned  to  one  group  or  the  other. 
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Each  mission  consisted  of  two  flight  segments  (CRUISE  and  CCIP  AUTO). 

The  data  being  reported  in  this  technical  report  was  collected  only  in 
the  CRUISE  segment.  After  each  pilot  performed  the  final  task  for  the 
CRUISE  segment,  the  CCIP  AUTO  segment  was  begun.  During  this  segment 
half  of  the  pilots  flew  a  modified  pop-up  maneuver  using  conventional  A-7 
like  HUD  symbology,  while  half  of  the  pilots  flew  the  same  maneuver 
using  an  experimental  integrated  HUD  symbology.  The  results  of  this 
segment,  along  with  a  follow-on  study  of  the  two  HUD  symbologies,  were 
reported  separately  (Reference  6).  Therefore,  the  CCIP  AUTO  segment 
will  be  mentioned  only  when  necessary  to  clarify  experimental  details. 

Each  of  the  four  data  missions  contained  the  same  number  and  types 
oftasks— nine  symbology  questions  and  six  MFK  tasks.  With  the  exception 
of  the  last  task  which  was  always  a  Weapons  Option  MFK  task  leading  into 
the  CCIP  AUTO  segment,  the  order  of  the  tasks  was  randomly  assigned. 

The  order  in  which  pilots  were  shown  situation  formats  for  each  symbol 
density  level  was  determined  by  the  use  of  a  trend-free  systematic  design 
(Reference  7).  This  method  is  appropriate  when  there  are  few  treatment 
levels  and  the  levels  are  administered  more  than  once  to  a  subject. 

3.  TEST  SUBJECTS 

A  total  of  eighteen  A-7  pilots  served  as  subjects  in  this 
experiment.  They  had  an  average  of  2305  flying  hours,  with  an  average 
of  493  hours  in  the  A-7D.  Half  of  the  pilots  flew  with  the  121st 
Tactical  Fighter  Wing  (ANG)  at  Rickenbacker  AFB  and  half  with  the  178th 
Fighter  Group  (ANG)  at  the  Springfield  Municipal  Airport,  Springfield, 
Ohio. 


4.  TEST  PROCEDURE 

a.  Home-Base  Pilot  Briefings 

Prior  to  the  experiment,  the  members  of  the  experimental  team 
traveled  to  each  of  the  Air  National  Guard  units  to  give  the  subjects  an 
Initial  briefing.  Throughout  the  two  briefings  the  procedures  were 
standardized  such  that  each  pilot  received  the  same  Information  and 
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opportunity  for  familiarization  with  the  purpose  of  the  experiment,  the 
procedures  that  would  be  used,  the  cockpit  simulator,  and  the  types  of 
symbology  questions  and  MFK  tasks  they  would  be  asked  to  perform.  The 
initial  briefing  sought  to  familiarize  the  pilots  with  the  relevancy  of 
the  experiment  with  regard  to  future  fighter  systems,  to  acquaint  them 
with  the  advanced  "digital"  aircraft  concept  and  the  simulator  cockpit 
controls  and  displays,  and,  finally,  to  introduce  them  to  the  Tailored 
Logic  used  to  complete  each  type  of  MFK  task. 

Following  the  group  briefing,  short  training  sessions  on  MFK 
Tailored  Logic  was  held.  Using  a  mockup  of  the  simulator  MFK  composed  of 
a  random  access  slide  projector  and  a  control  panel,  one  experimenter 
demonstrated  the  progression  of  logic  levels  for  each  type  of  task.  A 
briefing  package  illustrating  each  MFK  task  logic  sequence  was  given  to 
each  pilot  for  further  study. 

b.  On-Site  Pilot  Briefings 

After  the  home  base  training,  each  pilot  traveled  to  Wright- 
Patterson  AFB  for  the  on-site  cockpit  simulator  briefing,  training,  and 
testing.  The  Information  explained  or  demonstrated  during  the  briefing 
was  as  follows: 

1)  Unique  aspects  of  the  simulator  as  compared  to  the  A7 

2)  Flight  segments  and  the  HUD  symbology  to  be  flown 

3)  Flight  performance  measures  and  symbology  recognition 
performance  measures  being  taken 

4)  The  order  of  the  test  flights 

5)  HSF  format  (Including  combat  situation  symbology  and  the 
different  density  levels) 

6)  Procedure  for  answering  symbology  questions  and  examples 
of  each  type  of  question  (see  Figure  10) 

7)  MFK  and  DEK  operation  for  each  type  of  MFK  task 
(including  clear  function  and  correction  procedures  and  error  messages) 
(see  Table  1) 

i 

8)  Status  formats 
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9)  Use  of  the  throttle,  rudder  pedals,  trim  switch,  bomb  release 
button,  backup  flight  instruments,  flight  mode  switches  (inoperable), 
intercocnm  system,  brightness  controls,  master  arm  switch,  hydraulic  abort 
switch,  cockpit  exit  procedures,  flight  gloves,  and  flight  plans 
10)  CCIP  weapon  delivery  details. 

After  the  experimenter  concluded  the  briefing,  the  pilot  was 
allowed  to  practice  the  symbology  questions  and  MFK  tasks  while  the 
simulator  was  in  a  "hold"  condition.  Following  this  practice  session, 
the  pilot  was  given  a  few  minutes  to  practice  flying  the  simulator.  A 
fifteen-minute  break  followed  the  cockpit  briefing  and  pilot  practice 
session. 


c.  Training  Flights 

The  first  achromatic-coded  mission  was  preceded  by  a  20-minute 
achromatic  training  flight  containing  each  type  of  symbology  question 
and  MFK  task  to  be  given  in  the  data  missions.  A  similar  training  flight 
using  both  color-  and  shape-coding  was  given  before  the  first  color  data 
mission. 


d.  Data  Flights 

(1)  Mission  and  Tasks 

Four  missions  involving  simulated  flight  in  the  CRUISE 
segment  were  used.  (Additional  mission  informat ior.  and  the  initial 
conditions  for  one  of  the  missions  are  provided  in  Appendix  E.) 
Throughout  each  flight,  the  symbology  and  information  displayed  on  the 
HUD  and  HSF  were  dynamic  in  response  to  thrust,  bank,  yaw,  and  pitch 
inputs.  The  groundtracks  did  not  involve  any  turns  greater  than 
35  degrees,  nor  altitude  changes  greater  than  4,000  feet.  Information 
on  the  status  format  was  updated  in  response  to  aircraft  position  and 
data  inputs  on  the  MFK. 

The  pilot's  flying  task  during  the  CRUISE  portion  was  to 
maintain  a  true  airspeed  of  350  knots  and  to  keep  the  flight  director 
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symbol  centered  on  the  HUD.  The  symbology  questions  asked  involved 
either  counting  particular  symbols  on  a  format,  searching  for  the 
location  of  a  specific  symbol,  or  comparing  specified  symbols  in  two 
quadrants.  The  MFK  tasks  given  the  pilot  involved  either  communications, 
navigation,  or  stores  management  functions.  The  symbology  questions  and 
MFK  tasks  were  felt  to  be  typical  tasks  a  pilot  would  perform  during  a 
mission  in  a  single-seat  fighter  equipped  with  a  real-time  situation 
display.  The  fact  that  flying  performance,  symbology  question 
performance,  and  keyboard  operation  performance  were  all  to  be  recorded 
was  stressed  to  the  pilots. 

Each  mission  contained  nine  symbology  questions  and  six  MFK  tasks 
randomly  ordered  (see  Table  2).  Mission  scenarios  were  constructed 
around  each  set  of  randomized  tasks  in  order  to  provide  a  high  degree  of 
external  realism.  In  this  way,  the  task  orders  appeared  logical.  The 
task  instructions  were  given  over  the  headset  using  standard  controller 
terminology  (see  Appendix  F  for  example  of  a  mission  script  text). 

(a)  Symbology  Questions  and  Corresponding  Formats 

A  total  of  36  symbology  questions  were  written--two 
questions  per  situation  format.  Each  pilot  received  all  36  questions, 
and  so  was  exposed  to  each  of  the  18  formats  twice.  However,  the 
experiment  was  designed  such  that  a  pilot  received  each  format  only  once 
in  each  coding  condition.  Thus,  he  saw  each  format  once  in  the  morning 
and  once  in  the  afternoon.  Each  symbology  question  and  its 
corresponding  situation  format  were  viewed  in  both  the  achromatic  and 
color  conditions. 


During  each  mission  a  pilot  received  a  counting 
question,  a  comparing  question,  and  a  searching  question  at  each  of  the 
three  density  levels.  Across  a  two-mission  series  of  the  same  coding 
conditions,  the  pilot  received  each  one  of  the  above  questions  in  the 
simple  form  in  which  he  was  required  to  recognize  symbol  type  and  state. 
He  also  received  each  In  the  complex  form  in  which  he  was  required  to 
recognize  orientation  as  well  as  symbol  type  and  state  (see  Figure  10). 
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Task 

Number 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 


TABLE  2 

THE  SYMBOLOGY  QUESTIONS  AND  MFK  TASKS  (IN  ORDER) 
FOR  TWO  OF  THE  DATA  MISSIONS 


Mission  5 

Search  (simple)  for ^  s 

Count  (complex) A  heading  12  o'clock 


Mission  11 

Change  UHF  to  303.4 

Search  (simple)  for 
^  s 


Change  UHF  to  348.0 


Search  (simple)  for  s  s 

Compare  (complex)  quadrants  1  and  4 
for  most  A  heading  12  o'clock 


Compare  (complex)  quadrants  1  and  4 
for  most  A  heading  6  o'clock 


Squawk  IFF  Mode  3  in,  and  change 
code  to  3700 

Compare  (simple)  quadrants  3  and 
4  for  most1"#1 


Change  IFF  Mode  3  to  2100 
Change  TACAN  to  115X 

Search  (complex)  forT  heading 
12  o'clock 

FLY  TO  Waypoint  9 

Count  (simple) rA1s 

Count  (complex) /Si  heading  9  o'clock 

Select  Weapon  Option  2 


Search  (complex) 
for  A  heading 
6  o'clock 

Change  TACAN  to 
125Y 

Squawk  IFF  Mode  3 
in,  and  change 
code  to  3600 

Search  (complex) 
for'*'1  heading 
3  o'clock 

Count  (simple) A 


Compare  (complex) 
quadrants  1  and  3 
for  T1  heading 
3  o'clock 

Count  (complex)  A 
heading  9  o'clock 

Compare  (simple) 
quadrants  1  and  2 
for  A  s 

Change  IFF  Mode  3 
to  4100 

Compare  (simple) 
quadrants  1  and  2 
for  A 

Count  (simple)^ 

FLY  TO  Waypoint  6 

Select  Weapon 
Option  4 
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COUNT  QUESTIONS  I  TOTAL  DISPLAY  I 

•  HOW  MANY  A  ANE  THEM?  I  SIMPLE  I 

•  HOW  MANY  A  HEADING  AT  •  O'CLOCK?  ICOMPLEXI 

SEARCH  QUESTIONS  I  TOTAL  OISPLAV  I 

•  IN  WHICH  OUAORANT  IS  THE  LOCATED?  ISIMPLEI 

•  IN  WHICH  OUADRANT  IS  O.  HEADIHG  AT  12  O'CLOCK?  ICOMPLEXI 

COMPARE  QUESTIONS 

•  DOES  OUADRANT  1  OR  2  HAVE  MORE  ?,?  ISIMPLEI 

•  DOES  OUAORANT  2  OR  4  HAVE  MORE  m  HEADING  AT  3  O'CLOCK?  ICOMPLEXI 

Figure  10.  Types  of  Symbology  Questions 


(b)  MFK  Tasks  and  Tailored  Logic 

During  the  study  only  Tailored  MFK  Logic  was  used. 

In  Tailored  Logic  the  ten  most  likely  control  functions  to  be  used  in  the 
current  flight  mode  are  active  and  available  to  the  pilot  through  the 
five  push-button  multifunction  switches  on  each  side  of  the  MFK  CRT.  All 
of  the  MFK  tasks  corresponded  to  one  of  these  ten  available  functions. 

The  first  five  tasks  were  given  during  the  CRUISE  mode,  while  the  sixth, 
the  Weapon  Option  Selection  task,  occurred  at  the  beginnning  of  the  CCIP 
AUTO  mode  (see  Figure  6  and  11  for  the  CRUISE  and  CCIP  AUTO  MFK  first 
pages,  respectively).  Table  3  lists  the  six  types  of  MFK  tasks  and  the 
switch  hits  required  to  accomplish  each. 


Figure  11.  Multifunction  Keyboard  (MFK)  for  CCIP  AUTO  Mode 
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TABLE  3 


MFK  TASKS  AND  THEIR  OPERATING  SEQUENCES 


UHF  Frequency  Change. 

The  pilot  had  to  select  the  UHF  CHNG  multifunction 
switch  on  the  MFK,  four  digits  on  the  DEK,  then  ENTER  on  the 
DEK. 


IFF  Mode  3  Change. 

The  pilot  had  to  select  the  MODE  3  multifunction  switch 
on  the  MFK,  four  digits  on  the  DEK,  then  ENTER  on  the  DEK. 

IFF  Mode  3  Chanae  and  Mode  3  In. 


This  was  the  only  MFK  task  consisting  of  two  subtasks. 

Sub  task  a:  IFF  Mode  3  Change.  See  above  description 
of  Task  2. 

Subtask  b:  IFF  Mode  3  In.  The  pilot  had  to  select  the 
MODE  3  multifunction  switch  on  the  MFK,  then  ENTER  on 
the  DEK. 

TACAN  Channel  Change. 

The  pilot  had  to  select  the  TCN  CHNG  multifunction 
switch  on  the  MFK,  three  digits  and  one  letter  on  the  DEK, 
then  ENTER  on  the  DEK. 

FLY  TO  (Waypoint  Number 


The  pilot  had  to  select  the  FLY  TO  multifunction  switch 
on  the  MFK,  one  digit  on  the  DEK,  then  ENTER  on  the  DEK. 

Weapon  Selection. 

the  weapon  selection  MFK  task  was  always  the  last  task 
before  the  CCIP  AUTO  pop-up  maneuver.  Prior  to  the  weapon 
selection,  the  track  was  advanced  to  the  beginning  of  the 
second  flight  segment  and  the  CCIP  tailored  page  appeared  on 
the  MFK  (see  Figure  11). 

The  pilot  then  had  to  select  the  appropriate  WPN  OPTION 
multifunction  switch  on  the  MFK.  This  completed  the  task. 
However,  before  the  weapon(s)  could  be  released  the  pilot 
had  to  turn  on  the  Master  Arm  Switch. 
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(2)  Prodedures 

Two  of  the  three  experimenters  were  stationed  at  the 
experimenters'  console  inside  the  testing  room  and  monitored  pilot 
performance,  while  the  third  experimenter  was  stationed  at  the  computer 
terminal.  Experimenter  #1  was  tasked  with  initiating  pre-event  periods, 
initiating  and  controlling  task  sequences  and  data  recordings,  and 
terminating  training  and  data  flights.  Experimenter  #2,  an  experienced 
fighter  pilot,  gave  the  cockpit  briefing  and  demonstration  prior  to 
testing.  During  the  training  and  data  missions.  Experimenter  #2  played 
the  role  of  a  mission  controller--reading  from  a  scenario  script 
constructed  for  each  mission.  The  third  experimenter  was  responsible  for 
initiating  the  appropriate  missions,  entering  responses  to  symbology 
questions,  ensuring  data  recording  on  magnetic  tape,  and  generally 
monitoring  the  computer  operations.  Flow  diagrams  of  the  procedural 
steps  for  symbology  questions  and  MFK  tasks  discussed  in  the  following 
paragraphs  are  shown  in  Figures  12  and  13  respectively.  Basically,  the 
procedural  steps  were  broken  down  into  four  distinct  stages  or  periods: 
pre-event,  task  event  instruction,  task  event,  and  post-event.  During 
the  pre-event,  thirty  seconds  of  flight  performance  data  were  recorded. 
This  flight  performance  data  was  designated  as  baseline  performance,  and 
was  substracted  from  the  pilot's  flight  performance  data  during  the  task 
event.  This  difference  score  quantified  the  level  of  flight  performance 
during  the  task.  Details  of  how  this  was  done  are  discussed  in  the 
final  subsection  in  the  Section.  During  the  task  event  instruction 
period  the  pilot  received  verbal  instruction  from  Experimenter  #2  and 
responded  that  he  understood  the  task.  The  event  period  for  the  MFK 
tasks  consisted  of  the  tim4>it  took  the  pilot  to  correctly  perform  the 
task.  (Note  this  included  the  time  required  to  make  any  corrections.) 

For  the  symbology  questions,  the  event  period  covered  only  the  time  it 
took  the  pilot  to  give  his  first  response— whether  it  was  correct  or 
Incorrect.  A  fourth  period,  the  post-event,  existed  only  for  the 
symbology  questions.  During  this  time  period,  any  errors  made  were 
identified  and  the  correct  response  given  to  the  pilot. 
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(a)  Pre-event  Period  of  Baseline  Flight 

Performance  for  Symbology  and  MFK  Tasks 

Experimenter  #1  pushed  a  PRE-EVENT  switch  on  the 
console  which  started  a  thirty-second  timer.  Activation  of  this  switch 
automatically  designated  the  beginning  and  end  of  the  pre-event  period  of 
baseline  flight  performance.  The  PRE-EVENT  switch  remained  lighted 
during  the  pre-event  period.  Concurrently,  a  countdown  by  seconds  was 
displayed  on  the  experimenters'  status  display.  When  the  displayed 
countdown  reached  zero,  the  zero  flashed  until  Experimenter  #1  initiated 
the  task. 


(b)  Task  Event  Instruction  Period  for 

Symbology  and  MFK  Tasks 

Once  the  thirty  second  pre-event  period  of  baseline 
performance  had  been  recorded  (PRE- EVENT  switch  light  off,  countdown 
"zero"  flashing).  Experimenter  #2  read  either  a  symbology  question  to  the 
pilot  or  a  command  to  accomplish  an  MFK  task.  (The  experimenters 
avoided  initiating  tasks  when  the  pilot  would  have  to  complete  the  task 
while  flying  a  turn  on  the  groundtrack . )  Experimenter  #2  read  from  the 
written  script  to  ensure  the  pilots  received  identical  instructions  for  a 
particular  task  and  mission  (see  Appendix  F).  A  verbal  response  from 
the  pilot  was  required  to  ensure  that  he  understood  the  task.  Using 
these  procedures,  time  to  request  a  task  or  to  acknowledge  the 
instruction  was  not  a  part  of  the  pre-event  or  task  event  time. 

The  information  required  by  the  pilot  to  complete 
the  MFK  tasks  was  provided  on  modified  Flight  Plans  (AF  Form  70s)  and 
was  referenced  during  the  instructions  by  the  corresponding  letter,  or 
number  (see  Appendix  G  for  an  example  Flight  Plan).  The  following  were 
identified  by  a  letter  or  number  on  the  Form  70:  UHF  frequencies,  IFF 
codes,  altimeter  settings,  ILS  frequencies/courses,  waypoint  identifiers, 
and  TACAN  channels.  Weapon  parameter  information  was  identified  by 
weapon  option  numbers.  By  Identifying  the  information  in  this  way, 
errors  due  to  faulty  communication  or  forgetting  of  the  information  were 
minimized. 
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(c)  Task  Event  Period  and  Post  Period  for 

Symbology  Tasks 

Concurrent  with  the  pilot's  acknowledgement  of  the 
instructions.  Experimenter  #1  pushed  an  EVENT  MARKER  switch  on  the 
console  to  initiate  the  task.  If  the  task  was  a  symbology  question,  this 
switch  action:  1)  lit  up  the  EVENT  MARKER  switch;  2)  displayed  new 
format  appropriate  to  that  question  on  the  HSF;  3)  started  a  30-second 
timer  on  the  console  status  display;  and  4)  initiated  recording  of  flight 
parameters.  If  the  pilot  answered,  either  correctly  or  incorrectly, 
before  the  thirty  seconds  were  up.  Experimenter  #1  pushed  the  TASK  ABORT 
switch  on  the  console  immediately  to  record  response  time  and  terminate 
the  recording  of  flight  parameters.  Response  time  was  automatically  cut 
off  after  30  seconds.  Experimenter  #3  then  entered  the  pilot's  first 
response  through  the  terminal  after  the  thirty  seconds  were  up.  The 
EVENT  MARKER  switch  light  extinguished  upon  entry  of  the  pilot's  response 
through  the  terminal.  If  the  pilot  made  an  error.  Experimenter  #2 
informed  him  of  the  error  and  the  correct  response.  This  frequently 
prompted  the  pilot  to  provide  valuable  information  on  which  of  the 
symbol (s)  he  had  mistaken  or  which  symbol  location(s)  he  had  overlooked 
(see  pilot  comments  in  Appendix  H).  The  response  accuracy  was  also 
available  from  the  voice  recording  of  the  task  instructions  and  the 
pilot's  response.  The  tape  recording  of  the  testing  session  could  be 
used  to  verify  experimenter  recorded  data. 

(d)  Task  Event  Period  for  MFK  Tasks 

As  with  the  symbology  question  procedure. 

Experimenter  #1  pushed  the  EVENT  MARKER  switch  on  the  console  when  the 
pilot  acknowledged  the  task.  This  action:  1)  lighted  the  EVENT  MARKER 
switch;  2)  unlocked  the  cockpit  MFK;  3)  started  the  console  timer;  and 
4)  started  the  recording  of  flight  parameters  and  MFK  task  data.  The 
EVENT  MARKER  switch  light  went  out  after  successful  completion  of  the 
task  event  or  when  the  TASK  ABORT  was  selected.  Activation  of  the  TASK 
ABORT  switch  on  the  experimenters'  console  terminated  recording  of  the 
MFK  task  and  provided  the  experimenters  with  the  capability  to  Initialize 
the  pre-event  period  for  the  next  programmed  task.  This  switch  was  only 
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used  during  an  MFK  task  if  the  experimenters  determined  that  the  data 
being  recorded  was  unusable  and  that  the  task  would  eventually  have  to  be 
rerun.  Refer  to  Appendix  A  for  a  complete  description  of  the  MFK 
operating  procedures  and  control  logic.  For  a  listing  of  the  specific 
MFK  tasks  given,  see  Table  3. 

(e)  Mission  Completion 

After  the  pilot  successfully  completed  all  the 
required  tasks  for  the  flight.  Experimenter  #1  terminated  the  flight  by 
pushing  the  MISSION  COMPLETE  switch  on  the  console.  After  the  flight  had 
been  terminated,  the  summary  statistic  program  was  run  to  insure  that 
all  of  the  data  had  been  recorded.  If  the  simulation  facility  broke 
down  during  testing,  or  the  data  recording  malfunctioned,  the  capability 
existed  to  record  data  for  any  single  task  without  rerunning  the  whole 
data  flight.  The  scoring  program  recognized  only  completed  blocks  of 
data  and  ignored  any  incomplete  task  data. 

(f)  Pilot  Debriefing 

Immediately  after  the  two  consecutive  data  missions 
using  one  symbology  format  coding  condition,  the  pilots  were  given  a 
debriefing  questionnaire  concerning  that  coding  condition.  Following  the 
completion  of  all  data  flights,  pilots  were  given  a  final  overall 
debriefing  questionnaire,  eliciting  comparisons  of  the  two  types  of 
symbol  coding.  The  cover  page  of  the  final  debriefing  questionnaire  also 
contained  a  number  of  questions  about  the  pilot's  flying  experience  and 
background  (see  Appendix  H  for  all  three  debriefing  questionnaires,)./ 

(g)  Performance  Measures  and  Data  Analysis 

The  pilot's  performance  in  terms  of  flying  the 
simulator,  answering  the  symbology  questions,  and  performing  MFK  tasks 
was  measured.  For  the  purpose  of  this  report,  only  the  flight 
performance  data  and  symbology  question  data  were  analyzed. 


V 
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The  following  three  flight  parameters  were  recorded  two  times  per 
second  on  magnetic  tape: 


a)  True  airspeed  deviation  (knots): 

b)  Flight  director  horizontal  steering  error  (pixels);  and 

c)  Flight  director  vertical  steering  error  (pixels). 

Appropriate  summary  statistics  (average  error,  AE;  average  absolute 

error,  AAE;  root-mean-square  error,  RMS;  standard  deviation,  SO)  were 
computed  on  these  flight  parameters  for: 


a)  The  thirty  second  period  prior  to  each  task  event  (pre-event 
period);  and 


b)  The  time  period  between  EVENT  MARKER  switch  selection  and  TASK 
ABORT  switch  selection. 


The  thirty  second  pre-event  time  was  designated  as  baseline 
performance.  Summary  statistics  for  the  pre-event  time  for  each 
parameter  were  substracted  from  the  corresponding  values  computed  for 
the  time  period  required  by  the  pilot  to  complete  an  assignment  task 
event.  This  difference  score  quantified  the  flying  task  performance 
during  the  symbology  tasks. 


Symbology  task  performance  was  also  evaluated  by  measuring: 


a)  Response  time  for  each  of  the  questions;  and 

b)  Number  of  question  errors 


Two  of  the  question  types  required  the  pilot  to  respond  with  a 
quadrant  number  (search  and  comparison),  while  the  third  required  a 
numerical  answer  (count).  When  Experimenter  #3  entered  the  pilot's 
response  to  a  symbology  question,  the  computer  recorded  a  "i"  for  a 
correct  response  or  M-l“  for  an  incorrect  response. 


The  RMS  values  for  the  three  flight  performance  variables  and  the 
response  time  were  initially  analyzed  by  multivariate  analysis  of 
variance  (MANOVA)  using  the  SPSS  (Statistical  Package  for  the  Social 
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Sciences)  MANOVA  program  available  on  the  CDC  6600  Computer  System 
(Reference  8).  (Error  data  was  not  included  in  the  MANOVA  since  the 
computation  method  among  questions  types  differed.)  In  those  cases  where 
the  MANOVA  revealed  significant  effects,  subsequent  Finite  intersection 
Tests  (FITs)  were  conducted  in  order  to  determine:  1)  which  of  the 
dependent  variables  were  most  sensitive  to  changes  in  independent 
variables;  and  2)  which  of  the  experimental  groups  differed  significantly 
from  each  other  (Reference  9). 

Wilcoxin  matched-pairs  signed-ranks  tests  and  Friedman  two-way 
analyses  of  variance  (ANOVAs)  by  ranks  were  used  to  analyze  error  data 
(Reference  10). 

Data  obtained  from  the  debriefing  questionnaires  was  compiled  to  be 
presented  in  tabular  form  and  appropriate  nonparametric  analyses  were 
conducted.  Descriptive  statistics  were  computed  on  the  biographical  data 
obtained  from  the  flight  experience  questionnaire  to  obtain  an  overall 
sample  view  of  the  characteristics  of  the  pilot  sample.  Nonparametric 
Kolmogorov-Smirnov  tests  of  significance  were  used  to  analyze  the 
questions  involving  rating  scales  (Reference  10). 
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SECTION  IV 
RESULTS 

In  this  Section  the  results  of  the  data  analyses  of  the  objective 
performance  measures  will  be  presented.  (A  significance  level  of  «  * 

0.05  was  set  for  each  of  the  analyses.)  The  nonparametric  analyses  of 
the  subjective  questionnaire  data  will  be  presented  as  appropriate  in  the 
following  section. 

Performance  differences  between  the  two  coding  conditions  at  each 
of  the  three  symbol  density  levels  will  be  presented  first.  This  will  be 
followed  by  the  performance  differences  among  the  three  symbol  types  in 
each  of  the  three  symbol  states.  Last,  performance  differences  among  the 
three  task  types  and  differences  between  simple  and  complex  tasks  will 
be  reported. 

1.  PERFORMANCE  DIFFERENCE  BETWEEN  CODING  CONDITIONS  AT 

EACH  DENSITY  LEVEL 

Performance  differences  between  the  achromatic  shape-coding 
condition  and  the  redundant  color-  and  shape-coding  condition  were 
analyzed  for  each  of  the  three  density  levels:  10,  20,  and  30  symbols. 

The  MANOVA  examining  flight  performance  data  and  response  time  data 
found  significant  main  effects  for  both  coding  condition  and  density 
level,  F  (4,14)  =  17.40,  p  <;  0.00003,  and  F  (8,64)  =  6,93  p  <  0.00001  , 

r\ 

respectively.  A  significant  interaction  effect  was  also  found  between 
coding  condition  and  density  level,  £  (8,64)  =  2.38,  £  <  0.03.  The  FIT 
analyses  Indicated  that  response  time  was  the  performance  measure  most 
affected  by  varying  the  Independent  variables.  As  shown  in  Figure  14,  as 
the  density  level  of  the  formats  Increased,  the  mean  response  times  were 
increasingly  faster  with  the  redundant  color-coding  than  with  the  shape- 
coding  alone.  Specifically,  the  FIT  analyses  revealed  that  performance 
was  significantly  better  with  the  redundant  color-coding  at  all  three 
density  levels:  DL  10  JF  (1,99)  *  10.782,  £  <  0.003;  DL  20  ^F  (1,99)  = 
37.123,  £  <  0.003);  and  DL  30  j[F  (1,99)  =  33.747,  £  <  0.003).  A  FIT 
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analysis  found  no  significant  difference  between  redundant  color-coding 
at  density  level  30  and  shape-coding  at  density  level  10.  The  difference 
in  response  time  between  these  two  conditions  was  less  than  one  second. 

The  Wilcoxin  test  used  to  analyze  the  error  data  found 
significantly  fewer  errors  in  the  redundant  color  condition  at  density 
level  30  only,  T,  (11)  =  3.5,  £  <  0.05  (see  Figure  15).  Although  not 
significant,  there  were  also  fewer  errors  with  redundant  color-coding  at 
density  level  20.  At  density  level  10  there  was  an  error  reversal  where 
the  color-coding  condition  had  more  errors  than  the  shape-coding 
condition.  The  reasons  for  this  reversal  will  be  explored  in  the 
following  section. 

2.  PERFORMANCE  DIFFERENCES  BETWEEN  THE  SYMBOL  TYPES 

IN  EACH  STATE 

Performance  differences  for  each  of  the  three  types  of  symbols  (•  , 

S,  and  A)  under  each  of  the  three  states  (A,m,  andr>)  were  also 
examined.  As  in  the  previous  MANOVA (subsection  1  of  Section  IV, 
paragraph  2),  only  flight  performance  measures  and  response  times  were 
included  as  dependent  variables.  Significant  main  effects  were  found  for 
both  symbol  type  ££  (8,64)  =  2.52,  £  <  0.02),  and  symbol  state  XL  (8»64) 

=  2.25,  £  <  0.03).  In  addition,  a  significant  interaction  effect  was 
indicated  between  symbol  type  and  state,  £  (16,272)  =  3.17,  £<  0.00005.  As 
was  the  case  in  the  earlier  MANOVA,  response  time  was  the  performance 
measure  most  affected  by  varying  the  independent  variables. 

Figure  16  presents  the  response  time  for  each  symbol  type  as  a 
function  of  symbol  state.  The  symbol  combination  had,  by  far,  the 
longest  mean  response  time.  The  FIT  analyses  found  it  to  differ 
significantly  from  all  of  the  other  symbols  combined,  £  (1,153)  =  41.13, 

£  <  0.05,  and  from  its  nearest  neighbor  the  (the  symbol  with  the 
next  longest  mean  response  time),  £  (1,153)  =  4.66,  £  <  0.05. 
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Figure  16.  Response  Time  for  Each  Symbol  Type  as  a  Function 
of  Symbol  State 
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FIT  analyses  found  the  'T'  symbol  to  differ  significantly  from  all 
of  the  other  symbols  combined,  £,  (1,153)  =  10.08,  £  <  0.05.  An 
additional  analysis  found  the  only  symbol  combination  it  did  not  differ 
significantly  from  was  the  ^  . 

Threat  error  data  could  not  be  analyzed  by  parametric  tests  because 
there  were  unequal  numbers  of  observations  for  the  various  symbol  states 
and  types.  Table  4  gives  the  number  of  errors  for  each  state  and  type, 
the  number  of  observations,  and  the  percentage  of  errors.  Figure  17 
presents  the  information  graphically. 

3.  PERFORMANCE  DIFFERENCES  AMONG  THE  TYPES  OF  SYMBOLOGY 

QUESTIONS  AT  EACH  COMPLEXITY  LEVEL 

An  additional  MANOVA  was  run  comparing  flight  performance  and 
response  time  data  for  aircraft  counting  questions,  comparison  questions 
and  searching  questions  at  each  of  two  complexity  levels.  One 
complexity  level  called  the  simple  form  required  recognition  of  aircraft 
symbol  type  and  state,  while  the  second  level  called  the  complex  form 
required  recognition  of  aircraft  symbol  orientation  as  well. 

The  MANOVA  revealed  a  significant  main  effect  for  question  type,  £ 
(8,64)  =  5.12,  £  <  0.0006,  but  no  main  effect  for  complexity  level  (£  < 
0.07).  A  significant  interaction  effect  was  found  between  question  type 
and  complexity  level,  £  (8,64)  =  2.27,  p  <  0.03.  Follow-up  FIT  analyses 
revealed  response  time  to  be  the  performance  measure  that  differed  most 
among  the  question  types  (see  Figure  18).  Simple  count  questions  were 
found  to  take  significantly  longer  to  answer  than  both  simple  search 
questions,  £  (1,99)  *  22.29,  £  <  0.002,  and  simple  compare  questions,  £ 
(1,99)  =  16.48,  £  <  0.002.  Complex  count  questions  were  found  to  take 
significantly  longer  to  answer  than  complex  compare  questions  only,  £ 
(1,99)  =  15.06,  £  <  0.002. 
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TABLE  4 

ERROR  DATA  FOR  EACH  SYMBOL  TYPE  UNDER  EACH  SYMBOL  STATE 

PERCENTAGE 

NUMBER  OF  ERRORS  NUMBER  OF  OBSERVATIONS*  ERROR  RATE 
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Figure  18.  Response  Time  as  a  Function  of  Question  Type 
and  Question  Complexity  Level 
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Wllcoxin  tests  used  to  compare  errors  between  complex  and  simple 
questions  for  each  question  type,  revealed  a  significant  performance 
difference  on  counting  questions  only,  T  (10)  *  8.0,  £  <  0.05  (see 
Figure  19).  However,  a  problem  format  involving  one  of  the  simple  count 
questions  makes  any  interpretation  of  this  result  highly  speculative 
(see  subsection  1  of  Section  5  and  Figure  20). 


Figure  19.  Number  of  Threat  Errors  for  Each  Question  Type 
by  Question  Complexity  Level 
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SECTION  V 
DISCUSSION 

The  outline  of  this  Section  follows  that  of  Section  IV: 

1)  Effectiveness  of  redundant  color-coding; 

2)  Effectiveness  of  the  various  symbol  types  and  states;  and 

3)  Performance  differences  as  a  function  of  question  type  and 
complexity. 

Where  appropriate* subjective  data  from  the  debriefing  questionnaires 
will  be  presented  and  discussed. 

1.  EFFECTIVENESS  OF  REDUNDANT  COLOR-CODING 

In  this  particular  application,  the  addition  of  redundant  color¬ 
coding  to  shape-coded  symbols  was  extremely  successful  in  reducing 
operator  response  time.  The  overall  average  response  time  for  a 
symbology  question  was  reduced  by  forty  percent  through  the  additon  of 
color-coding.  In  addition,  the  effectiveness  of  redundant  color  was 
more  pronounced  as  the  combat  situation  display  became  more  complex; 
i.e.,  as  the  number  of  symbols  on  the  display  increased.  Notice  in 
Figure  14  how  the  response  time  with  10  achromatic  symbols  is  almost 
identical  to  the  response  time  with  30  color  symbols.  This  reduction  in 
seconds  required  to  intepret  and  respond  to  a  real-time  situation 
display  is  critical  to  the  pilot  in  the  combat  situation.  Seconds  may 
save  his  life  and  aircraft.  In  addition,  the  seconds  saved  allow  him  to 
spend  more  time  "heads  up"  scanning  the  outside.  This  is  vital  when 
flying  low-level  at  high  speeds  and  attempting  to  outmaneuver  enemy  air¬ 
craft  or  jinking  to  avoid  SAM  or  AAA  fire. 

The  subjective  data  buttressed  the  objective  data.  On  the  final 
debriefing  questionnaire,  the  pilots  were  asked,  for  each  density  level, 
how  they  felt  the  addition  of  color-coding  affected  their  response  time. 
At  all  three  density  levels,  the  overwhelming  response  was  "decreased  It 
a  great  deal"  or  "decreased  It  some"  _Q>  <  .01  for  each  level).  The 


following  comments  were  typical  of  others  written  in  on  the 
questionnaires:  "[black-and-white]  as  opposed  to  the  color-coded  ones 
took  more  time";  "without  color,  ...  becomes  twice  as  difficult";  "the 
more  symbols  the  more  color  coding  helped."  On  the  final  debriefing 
questionnaire,  they  were  also  asked  to  rate  how  helpful  such  a  display 
would  be  in  an  actual  combat  mission.  For  the  color-coding  condition, 
all  respondents  checked  the  display  would  have  “helped  a  great  deal"  or 
"helped  some"  <  .01).  For  the  shape-coding  condition,  respondents 
were  pretty  evenly  split  as  to  whether  it  would  help  or  hinder.  The 
following  comment  states  the  ambivalence  well:  "Using  black  and  white 

with  ten  [symbols]  helped  some.  More  than  ten  would  hinder." 

'/>  h 

The  results  of  the  error  data,  as  shown  in  Figure  15,  is  a  little 

less  straight-forward.  There  were  significantly  fewer  errors  with 

color-coding  at  density  level  30  ^|>  <0.05),  and,  although  not 

significant,  there  were  fewer  errors  at  density  level  20.  One 

intriguing  feature  of  the  data  is  the  dramatic  decrease  in  number  of 

errors  in  the  color  condition  at  density  level  30.  It  is  hypothesized 

that  what  happened  is  that  pilots  changed  their  strategy  with  density 

level  30  formats.  Pilot  comments  on  the  debriefing  questionnaires  and 

comments  made  to  experimenters  after  the  experiment  suggested  that  when 

both  color-  and  shape-coding  were  available,  the  pilots  used  only  the 

color  to  determine  state.  (Typical  comments  were:  "Just  looked  for 

color  then  looked  to  see  if  SAM,  aircraft,  or  AAA  by  symbol";  "Suggest 

you  drop  [state]  symbols  when  using  color  and  use  only  basic  symbols 
* 

such  as  A,  S,  and  •.)  Looking  at  the  number  of  errors  at  density  levels 
10  and  20  suggests  this  strategy  might  have  been  used.  It  is 
hypothesized  that  with  the  density  level  30  formats  the  pilots  realized 
that  they  would  have  to  use  both  color-  and  shape-coding  to  pick  out  the 
appropriate  symbols  because  the  formats  were  so  complex.  Therefore, 
even  though  the  response  time  increased,  the  error  rate  decreased  because 
they  examined  each  symbol  more  carefully. 

Another  interesting  twist  to  the  error  data  was  the  unexpected 
directional  difference  at  density  level  10.  This  can  be  attributed 
primarily  to  one  specific  symbology  question  and  its  corresponding 
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format  (see  Figure  20).  The  question  was  a  counting  question:  "How 
many  unknown  aircraft  are  there?"  The  correct  answer  was  two.  None  of 
the  subjects  who  received  the  question  in  the  achromatic  condition 
missed  it.  However,  four  subjects  who  received  the  question  in  the 
color  condition,  saw  only  one  of  the  two  unknown  aircraft.  Two  of  these 
four  subjects  later  stated  over  the  headset  that  they  had  overlooked  the 
symbol  in  the  upper  part  of  quadrant  one.  Although  the  symbol  was 
moving  up  toward  the  two  lines  of  alphanumerics,  it  remained  clearly 
visible  on  the  screen  for  the  duration  of  the  task.  This  might  suggest 
that  when  color  was  present,  the  shape  of  the  hat  was  not  attended  to  at 
first  glance  but  noticed  somewhat  later.  Because  of  the  proximity  of 
the  yellow  symbol  to  the  white  alphanumerics,  the  yellow  symbol  appeared 
lighter  to  the  pilots  and  was  considered  part  of  the  white 
alphanumerics.  One  pilot  commented  on  the  color-coding  debriefing  ques¬ 
tionnaire  that,  "Yellow  was  a  little  difficult  when  it  was  close  to  the 
edge"  [where  the  white  alphanumeric  scale  readout  appeared].  Since 
this  particular  format  had  a  density  level  of  10,  ,i  tends  to  strengthen 
the  earlier  hypothesis  that  at  density  levels  10  and  20,  color  was 
scanned  for  first  on  the  display. 

a.  Results  Related  to  Color-Coding  Literature 

These  results  confirm  a  number  of  findings  in  the  current 
literature.  First,  redundant  color  with  shape-coding  decreases  the 
amount  of  response  time  in  search  tasks  (References  1,  3  and  5)  and  in 
identification  tasks  (References  2,  3,  and  5).  Thus,  in  displays  where 
response  time  is  of  utmost  importance,  redundant  color-  and  shape-coding 
has  a  high  pay-off. 

Additionally,  color-coding  appears  most  beneficial  for 
displays  which  are  unformatted,  displays  on  which  symbol  density  is 
high,  and  displays  on  which  color  coding  is  logically  related  to  the 
operator's  task  (References  5  and  11). 
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Figure  20.  Threat  Display  Format  1  (Density  Level  10) 
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2.  EFFECTIVENESS  OF  VARIOUS  SYMBOL  TYPES  AND  STATES 

Two  symbol  type/state  combinations  required  significantly  longer 
times  for  identification  than  all  other  combinations:  the  and  the 
^ .  Although  the  shape  of  these  two  symbol  combinations  is  distinctly 
different,  in  both  cases  the  top  of  the  letter  is  approximately  parallel 
with  the  hat-shaped  designator.  Thus,  the  similarity  in  hat-shaped 
designators  to  the  letter  shape  and  the  fact  that  the  designators  were 
so  close  to  the  letters  may  have  caused  them  to  be  more  easily 
overlooked  at  first  glance. 

Another  problem  concerning  the  use  of  the  'ft  symbol  combination 
surfaced  in  the  remarks  sections  of  the  debriefing  questionnaires. 

Several  pilots  indicated  that  the  ft  symbol  closely  resembled  a  delta¬ 
shaped  symbol  (>S)  that  has  occasionally  been  used  in  the  past  to  denote 
an  aircraft.  This  resemblance  may  have  created  confusion  leading  to  a 
longer  response  time. 

In  addlton  to  objecting  to  the  use  of  the  ft  symbol  combination, 
several  pilots  disliked  the  use  of  the  symbol  A  for  anti-aircraft 
artillery  regardless  of  the  state  designator.  They  claimed  to  associate 
the  symbol  A  with  aircraft.  One  pilot  suggested  that  to  avoid  this  mis¬ 
taken  association,  the  symbol  G  (l.e.,  gun)  be  used  for  anti-aircraft 
artillery  Instead.  Although  this  might  give  rise  to  a  "parallelism" 
problem  in  the  existing  friendly  state  this  would  not  be  a  problem 
If  the  A  and  /^designators  are  changed  as  recommended  in  the 
succeeding  paragraph. 

Error  rates  among  the  symbol  combinations  could  not  be  analyzed 
statistically  because  of  unequal  numbers  of  observations.  However,  the 
subjective  data  revealed  the  pilots  did  experience  some  difficulty  in 
differentiating  the  state  designators  with  the  moving  aircraft  symbols 
(•).  On  the  shape-coded  debriefing  questionnaire,  a  large  number  of 
the  pilots  checked  they  had  difficulty  differentiating  between  theft  andT^i 
symbol  combinations  in  all  four  orientations:  ft  vs  (14  pilots); 
vs  £)  (15);  ^  vs  (12);  <•  vs  (•  (14).  As  one  of  the  pilots 
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commented  on  this  questionnaire,  "In  black  and  white,  the  spear-shaped 
symbols  and  the  half-moon  shapes  were  difficult  to  distinguish."  Based 
on  this  confusion  factor  between  the  A  and  O  designators,  and  the 
previously  mentioned  problems  concerning  the  A  and  ^  it  is  suggested 
that  new  state  designators  be  designed  for  the  hostile  and  friendly 
states. 

3.  PERFORMANCE  DIFFERENCES  AS  A  FUNCTION  OF 

QUESTION  COMPLEXITY  AND  TYPE 

Two  important  concerns  in  the  design  of  a  real-time  situation 
display  symbology  set  are  how  much  information  can  be  conveyed  by  a  par¬ 
ticular  symbol  and  how  should  that  information  be  conveyed  by  a 
particular  symbol. 

In  this  study,  complex  questions  required  noting  three  encoded  var¬ 
iables  (type,  state,  and  orientation)  while  simple  questions  required 
noting  only  two  (type  and  state).  The  fact  that  the  complex  questions 
did  not  take  signif icantly  longer  to  respond  to,  and  the  fact  that  the 
complex  questions  did  not  have  significantly  greater  error  rates, 
suggests  that  encoding  three  variables  was  acceptable. 

The  significantly  longer  response  time  for  counting  questions  may 
be  accounted  for  by  the  fact  that  the  pilot  had  to  do  an  exhaustive  scan 

of  the  whole  display.  For  comparison  questions  he  could  zero  in  on  two 

specific  quadrants  of  the  display,  and  for  the  search  questions  he  had 
to  scan  only  until  he  found  the  appropriate  symbol.  In  interpreting  the 
greater  errors  in  counting  questions,  it  must  be  kept  in  mind  that  four 
of  those  errors  were  related  to  the  one  problem  question  discussed 
earlier  in  subsection  1  of  Section  V.  The  pilots  were  asked  on  the 
shape-coding  debriefing  questionnaire  to  rate  the  difficulty  of  the 
three  types  of  questions.  The  search  questions  were  rated  significantly 

easier  than  the  other  two  <  0.05). 

Relating  this  back  to  the  current  literature,  color  has  been  shown 
to  be  the  best  coding  method  when  search  tasks  were  to  be  given,  but 
numerals  were  found  to  be  the  best  coding  method  for  counting  and 
comparing  tasks  (References  11  and  12). 
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4.  PILOT  ACCEPTABILITY  OF  SYMBOL  ORIENTATION  TO  DENOTE  HEADING 

The  pilots  were  asked  specifically  in  the  shape-coding  debriefing 
questionnaire  to  rate  the  adequacy  of  aircraft  symbol  orientation  to 
denote  heading.  A  significant  number  of  them  rated  this  convention  as 
either  good  or  excellent  (D  =  0.489,  £  <0.01). 
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SECTION  VI 

I 

’  CONCLUSIONS  AND  RECOMMENDATIONS 

Based  on  the  evaluation  of  the  symbology  set  in  this  simulation 
study,  the  following  conclusions  and  recommendations  are  made: 

1.  Conclusion:  Redundant  color-coding  for  the  shape-coded  symbols  on 
a  situation  display  significantly  reduces  pilot  response  time. 

Recommendation:  Color  displays  and  redundant  color-coding  should 
be  used  for  any  real-time  combat  situation  symbology  display  installed  in 
fighter  aircraft.*" 

Recommendation:  Color  di;plays  and  redundant  color-coding  should 
be  incorporated  for  any  display  to  be  used  by  a  highly-loaded  operator 
in  the  performance  of  a  series  of  complex  tasks. 

2.  Conclusion:  The  ^  proved  to  be  a  poor  symbol  choice  in  terms  of 
slow  response  time  and  high  percentage  of  errors. 

Recommendation:  The  symbol  for  anti-aircraft  artillery  should  be 
changed  from  A  to  G  and  tested  in  future  simulations. 

3.  Conclusion:  Pilots  took  longer  to  respond  to  the  'ft  and  ^ 
symbols,  and  in  general  found  the  A  and  Estate  designators  easily 
confused. 

Recommendation:  A  new  hostile  (A)  state  designator  and  friendly 
ir)  state  designator  be  designed  and  tested  in  future  simulations. 

*N0TE:  Current  state-of-the-art  technology  has  not  produced  a  full- 
color  cathode  ray  tube  that  does  not  wash  out  in  high  ambient  light 
conditions  found  in  the  fighter  cockpit.  This  technology  should  be 
given  strong  military  research  and  development  support  in  order  that 

such  technology  will  be  available  in  time  for  implementation  in  the  next 
generation  fighter. 
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4.  Conclusion:  Counting  tasks  took  the  pilots  longer  to  complete,  and 
they  made  more  errors  than  on  the  other  two  types  of  tasks. 

Recommendation:  The  use  of  subscripts  to  denote  several  of  one 
symbol  type  rather  than  separate  symbols  be  examined  in  future 
simulations. 

5.  Conclusion:  Pilots  were  receptive  to  the  idea  of  using  aircraft 
symbol  orientation  to  denote  aircraft  heading. 

Recommendation:  The  convention  of  using  aircraft  symbol  orien¬ 
tation  to  denote  aircraft  heading  for  all  twelve  clock  positions  be 
tested  in  future  simulations. 

Recommendation:  The  simulation  incorporate  changes  of  aircraft 
heading  so  that  symbol  rotation  effects  can  be  measured. 

6.  Conclusion:  This  evaluation  relied  heavily  on  the  subjective 
debriefing  questionnaire  data  to  interpret  the  objective  findings. 

Recommendation:  Any  studies  of  symbology  sets  to  be  used  by 
highly-loaded  operators  performing  complex  series  of  tasks  should  be 
evaluated  in  a  simulated  environment,  and  subjective  operator  data  should 
be  used  in  the  interpretation  of  the  objective  data. 
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APPENDIX  A 
COCKPIT  SIMULATOR 


1.  ELECTRO-OPTICAL  FORMATS 

Four  electro-optical  formats  were  used  in  the  present  study  to 
provide  information  for  utilization  by  the  pilot  (see  Figure  2).  The 
following  describes  each  display  in  detail: 

a.  Head  Up  Display  (HUD)  (see  Figure  3) 

The  HUD  supplied  flight  information  in  symbolic  form  on  a 
combiner  glass  in  the  pilot's  forward  field  of  view.  The  horizon  and 
flight  path  angle  lines  of  the  flight  path  scale  represented  the  horizon 
and  each  five  degrees  of  flight  path  angle  (FPA)  between  plus  and  minus 
90  degrees.  Positive  FPA  was  presented  as  solid  lines  and  appeared 
above  the  horizon  line.  Negative  FPA  was  presented  as  dashed  lines  and 
appeared  below  the  horizon  line.  The  five  degree  increments  were 
numbered  on  either  end  of  the  FPA  lines.  A  minus  sign  preceded  the 
numbers  for  negative  angles. 

The  aircraft  velocity  vector  was  represented  by  a  flight  path 
marker  (FPM)  which  denoted  the  point  toward  which  the  aircraft  was  flying 
at  all  times.  The  FPM  moved  horizontally  and  vertically,  but  was  not 
roll  stabilized  to  show  bank  angle.  Rather,  the  flight  path  scales  and 
their  associated  numbers  were  roll -stabilized  and  rotated  to  the 
appropriate  bank  angle. 

The  airspeed,  heading,  and  altitude  scales  were  not  roll- 
stabilized.  The  airspeed  and  altitude  scales  were  vertical  and  appeared 
on  the  left  and  right  sides  of  the  display,  respectively.  The  heading 
scale  was  horizontal  and  appeared  at  the  top  of  the  display.  The 
airspeed  scale  was  graduated  in  25  knot  increments  and  numbered  each  50 
knots.  At  least  three  sets  of  numbers  were  visible  at  all  times.  An 
exact  readout  of  current  airspeed  was  presented  in  the  window  in  the 
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center  of  the  scale.  The  readout  changed  whenever  the  airspeed  changed 
by  one  knot.  When  any  digit  changed  faster  than  two  times/second,  that 
digit  was  displayed  as  zero.  The  scale  moved  continuously.  The  scale 
numerics  were  not  superimposed  over  the  window,  but  in  that  area  were 
removed  from  the  CRT  by  software  blanking.  The  abbreviation  TAS  for 
true  airspeed  appeared  below  the  airspeed  scale  in  the  CRUISE  segment. 

The  current  UHF  radio  frequency  was  displayed  below  the  airspeed  scale. 

Barometric  altitude  was  displayed  on  the  altitude  scale  on  the 
right  side  of  the  HUD.  The  scale  was  graduated  in  250-foot  increments 
numbered  each  500  feet.  At  least  three  sets  of  numbers  were  visible  at 
all  times.  The  total  range  of  the  altitude  scale  was  from  minus  1,000 
feet  to  plus  99,999  feet  with  1,500  feet  in  view  at  all  times.  An  exact 
readout  of  the  altitude  was  provided  in  the  window  in  the  center  of  the 
scale.  The  readout  changed  whenever  the  altitude  changed  by  one  foot. 

The  scale  moved  continuously.  The  scale  numerics  were  not  superimposed 
over  the  window,  but  were  removed  from  the  CRT  by  software  blanking  in 
that  area.  When  a  500-foot  scale  mark  moved  off  the  scale,  the  numerics 
were  removed  at  that  end.  Numerical  digits  were  added  to  the  scale  when 
a  500-foot  mark  was  added  to  the  scale  as  it  moved.  When  any  digit 
changed  faster  than  two  times/second,  that  digit  was  displayed  as  zero. 

An  abbreviation  designating  the  current  flight  segment  appeared  below  the 
altitude  scale.  The  abbreviation  CRUS  was  displayed  during  the  CRUISE 
flight  segment. 

The  heading  scale  was  displayed  at  the  top  of  the  HUD. Forty  scale 
degrees  were  in  view  at  all  times,  graduated  in  five  degree  increments. 
Every  other  increment  was  labeled  by  a  two  digit  number  above  it.  Total 
heading  scale  range  was  360  degrees.  The  aircraft  magnetic  heading  was 
displayed  to  the  nearest  degree  in  the  window.  The  scale  moved 
continuously.  When  a  ten  degree  mark  moved  out  of  the  f ield-of-view,  two 
digits  were  removed  at  that  end.  Two  digits  were  added  to  the  scale 
when  a  ten  degree  mark  was  added  to  the  scale.  The  scale  numerics  were 
not  superimposed  over  the  window,  but  were  removed  from  the  CRT  by 
software  blanking  in  that  area. 
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The  flight  director  symbol  indicated  horizontal  and  vertical 
steering  error  information  with  respect  to  the  flight  path  marker.  The  X 
and  Y  commands  to  position  the  flight  director  symbol  were  such  that  the 
pilot  flew  the  flight  path  marker  to  the  flight  director  by  steering  the 
aircraft  in  pitch  and/or  bank  angle,  i.e.,  the  flight  director  was  moved 
by  the  software  to  the  flight  path  marker  when  the  proper  control 
signals  were  received. 

The  vertical  velocity  was  displayed  above  the  altitude  scale  in 
digital  form  with  the  readout  changing  in  one  foot  per  minute  increments 
over  a  range  of  0  to  9,999  feet  per  minute.  A  caret  indicated  vertical 
velocity  direction,  i.e.,  up  or  down.  As  in  the  altitude  scale,  when 
any  digit  on  the  format  changed  faster  than  two  times  per  second  that 
digit  was  displayed  as  zero. 

b.  Horizontal  Situation  Format  (HSF)  (see  Figure  4) 

A  seven-inch  diagonal  color  monitor  was  used  to  present 
simplified  navigation  information  in  a  track-up  format.  A  map 
representing  a  combat  environment  80  nautical  miles  in  length  was  used. 

The  aircraft  symbol  was  fixed  in  the  center  of  the  display.  The  aircraft 
track  was  displayed  at  a  fixed  location  centered  at  the  top  of  the  HSF. 
The  value  displayed  ranged  from  0  to  359  degrees.  The  track  to  the  next 
succeeding  waypoint  was  displayed  in  the  lower  left  corner  as  indicated 
in  the  figure  with  a  N  preceding  the  numerical  value  of  the  track.  The 
map  scale  was  displayed  in  the  lower  left  corner  of  the  HSF  just  below 
the  track  of  the  next  waypoint.  This  scale  indicated  the  range  covered 
by  the  map  in  nautical  miles  measured  vertically.  The  fuel  quantity  was 
displayed  in  a  fixed  location  to  the  right  of  the  range  scale  in  the 
lower  portion  of  the  HSF.  Fuel  quantity  was  displayed  with  an  F 
preceding  the  digital  readout  of  the  remaining  fuel  in  pounds,  e.g., 

F  17500. 
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The  waypoint  symbol  and  its  identification  (ID)  were  displayed 
anytime  that  waypoint  was  on  the  map.  Each  waypoint  had  an  ID,  which  was 
its  numerical  designation.  Information  concerning  the  next  waypoint, 
its  distance  from  the  aircraft,  and  the  time  required  to  reach  it  was 
displayed  in  the  upper  left  portion  of  the  HSF.  The  waypoint  identifier 
was  given,  followed  by  the  distance  in  nautical  miles  and  the  time  to  go 
to  the  nearest  tenth  of  a  minute. 

The  distance  and  time  to  the  next  target  (along  the  planned  route) 
were  displayed  in  the  upper  right  portion  of  the  HSF.  Again  the  target 
identifier  was  given,  followed  by  the  distance  in  nautical  miles,  and  the 
time  to  go  to  the  nearest  tenth  of  a  minute.  The  track,  distance,  and 
time  required  to  fly  directly  to  the  next  target  were  displayed  in  the 
lower  right  position  of  the  HSF  under  abbreviations  DIR  TO  TGT. 

The  groundspeed  was  displayed  above  the  time  to  the  next  target. 
Display  of  groundspeed  was  to  the  nearest  knot  and  was  preceded  by  the 
alphabetical  characters  GS,  e.g.,  GS  461. 

The  crosstrack  deviation  was  displayed  on  the.  HSF  by  the  relative 
displacement  of  the  map  track  line  from  the  aircraft  symbol.  The 
aircraft  was  positioned  in  the  center  of  the  display  and  the  map  moved 
under  the  aircraft  symbol  and  was  positioned  to  show  the  actual  aircraft 
position  in  relation  to  the  desired  track. 

The  symbology  formats  were  overridden  by  the  aircraft  symbol,  its 
groundtrack,  and  the  areas  at  the  top  and  bottom  of  the  display  dedicated 
for  alphanumerics. 

r 

/ 

c.  Multifunction  Keyboard  (MFK)  (see  Figures  6  and  11) 

Each  MF*  task  event  consisted  of  either  one  task  or  several 

subtasks.  A  sabtask  is  defined  as  a  set  of  MFK  and  DEK  selections  which 

» 

/ 

could  be  considered  a  complete  task  if  accomplished  independently.  The 
only  MFK  task  event  given  in  this  study  that  consisted  of  two  or  more 
subtasks  was  the  IFF  mode/code  change.  The  pilot  had  to  accomplish  both 
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a  mode  change  and  a  code  change  in  order  to  complete  one  task  event.  The 
pilot  was  allowed  to  accomplish  either  the  mooe  change  or  code  change 
first. 

Due  to  computer  memory  limitations  and  time  constraints,  only  the 
options  required  for  the  tasks  to  be  used  in  this  experiment  were 
programmed.  If  the  pilot  selected  an  option  that  was  not  programmed,  he 
received  the  message  OPTION  N/A  for  that  switch  on  the  CRT.  The  legend 
disappeared  with  the  selection  of  a  programmed  option.  Within  the 
CRUISE  segment,  mistakes  made  by  pushing  an  inappropriate  programmed 
switch  were  also  corrected  by  selecting  the  correct  option. 

Once  the  pilot  had  progressed  to  the  switch  action  that  activated 
the  DEK  (DEK  illuminated),  each  digit  selected  was  displayed  to  the  pilot 
on  the  keyboard  CRT.  When  the  pilot  selected  the  last  digit  (defined  by 
the  type  of  task),  the  preentry  readout  provided  the  pilot  with  the 
capability  to  verify  that  the  digits  selected  were  accurate.  If  the 
pilot  made  an  error  that  was  in  the  appropriate  range  or  realistic  for 
the  task  (example:  236.7  instead  of  236.6  UHF  as  the  frequency),  the 
incorrect  digit(s)  were  displayed  in  the  pre-entry  readout  (236.7).  In 
order  to  correct  the  mistake,  the  pilot  had  to  clear  the  incorrect 
digit(s).  One  push  of  the  CLEAR  key  on  the  DEK  erased  the  last  selected 
digit.  Two  pushes  of  the  CLEAR  key  erased  all  the  digits  selected  since 
the  last  activation  of  the  ENTER  key.  The  DEK  remained  activated  and 
lighted  after  any  push  of  the  CLEAR  key. 

In  addition  to  the  pre-entry  readout,  an  error  message  was 
displayed  to  the  pilot  when  an  error  was  made  that  was  out  of  the 
appropriate  range  or  unrealistic  for  a  task.  For  example,  if  9  was 
selected  for  the  first  IFF  code  digit  for  mode  3,  the  message  BAD  DATA 
was  displayed  to  the  pilot  next  to  the  pre-entry  readout  since  the  first 
digit  could  be  no  larger  than  7.  The  actual  illegal  digit  never 
appeared  on  the  pre-entry  readout,  but  was  ignored  by  the  computer.  The 
DEK  remained  active  and  when  the  pilot  made  a  legal  switch  hit  on  the 
DEK,  the  BAD  DATA  message  disappeared.  A  second  example  involves  a  pilot 
who  had  selected  69  as  the  first  two  digits  of  a  IFF  code  (6600)  instead 
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of  66.  In  this  case  the  first  digit  was  legal,  but  the  second  digit  was 
out  of  the  appropriate  IFF  range.  Since  the  computer  ignored  the 
illegal  digit  and  the  first  digit  selected  was  legal,  the  pre-entry 
readout  was  6  with  the  BAD  DATA  message  also  displayed.  Of  the  first  two 
digits,  only  the  second  digit  had  to  be  reselected. 

In  the  case  where  the  pilot  has  pushed  too  many  legal  digits,  the 
message  CHECK  OATA  was  displayed  next  to  the  pre-entry  readout,  the  pre¬ 
entry  readout  remained  except  that  the  surplus  digits  were  automatically 
ignored  by  the  computer,  and  the  DEK  remained  active.  If  the  remaining 
selected  digits  were  the  desired  entry  (6600  for  the  mode  3  code),  the 
pilot  pushed  the  ENTER  key.  If  the  desired  entry  was  6606  instead  of 
6600,  however,  the  pilot  had  to  operate  the  CLEAR  function  to  erase  the  6 
and  select  0  in  order  to  complete  the  task  event  correctly.  The  CHECK 
DATA  message  disappeared  with  the  first  hit  of  the  CLEAR  or  ENTER  key. 

Once  a  task  or  subtask  was  completed,  whether  it  was  correct  or 
incorrect,  all  recording  of  data  stopped.  If  a  data  entry  had  been 
required  as  part  of  the  task,  the  DEK  deactivated  and  the  pre-entry 
readout  disappeared.  The  computer  then  checked  to  see  if  the  data 
selections  and  entry  were  the  same  as  the  information  programmed  for  the 
subtask.  The  following  describes  the  MFK  configuration  and  operating 
procedures  after  the  computer  determined  whether  the  completed  task  or 
subtask  was  incorrect  or  correct. 

(1)  Incorrect  Task  or  Subtask  Completion 

If  a  task  or  subtask  was  completed  incorrectly,  the  pilot 
was  required  to  redo  it.  In  the  case  of  the  IFF  mode/code  change,  the 
pilot  only  had  to  redo  the  subtask  that  had  an  error.  The  subtask  or 
task  error  was  displayed  to  the  experimenters  on  the  experimenters' 
console  and  the  pilot's  MFK  became  inoperable.  Note  that  the  DEK  was 
deactivated;  if  the  pilot  had  to  make  a  digit  entry  to  correctly  complete 
the  retry,  he  had  to  reselect  the  switch  on  the  MFK  which  calls  up  the 
DEK.  After  thq  pilot  was  notified  by  Experimenter  #2  that  an  error  was 
made  and  the  pilot  responded  that  he  understood,  Experimenter  #1  selected 
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the  KEYBOARD  UNLOCK  switch  to  restore  the  MFK.  The  pilot  then  started 
to  redo  the  task  or  subtask;  the  pilot's  first  switch  hit  on  the  MFK  of 
the  retry  initiated  the  recording  of  data.  When  the  keyboard  operation 
was  completed  again,  the  recording  stopped  and  the  computer  verified  the 
entry. 

(2)  Correct  Subtask  or  Task  Completion 

When  the  computer  verified  that  a  completed  subtask  or 
task  was  correct,  the  computer  then  checked  whether  more  subtasks  were  to 
be  completed  at  that  time.  If  another  subtask  was  to  be  completed,  the 
MFK  remained  activated  at  the  last  level  used  during  the  completion  of 
the  previous  subtask.  (The  DEK  was  automatically  deactivated  at  subtask 
completion  whether  correct  or  incorrect.)  The  pilot's  first  switch  hit 
on  the  MFK  for  the  next  subtask  initiated  data  recording.  In  the  case 
where  no  more  subtasks  were  to  be  completed,  the  MFK  became  inoperable 
and  the  DEK  became  deactivated  and  the  task  was  considered  finished. 

An  exception  where  the  pre-entry  readout  terminal  readout 
remained  after  subtask  completion  was  when  the  pilot  entered  too  few 
legal  digits  (example:  236  for  236.7  UFH  frequency).  In  addition,  the 
MFK  and  DEK  remained  active  and  the  message  RE-ENTER  DATA  was  displayed 
next  to  the  readout.  The  pilot's  first  MFK  or  DEK  switch  hit  of  the 
retry  initiated  the  recording  of  data  and  erased  the  message. 

d.  Status  Format  (SF)  (see  Figure  5) 

A  six-inch  diagonal  black-and-white  monitor  was  used  to 
display  mission-related  status  information  to  the  pilot.  In  the  CRUISE 
segment,  the  SF  displayed  communications  and  navigation  data. 

2.  DEDICATED  DISPLAYS  AND  CONTROLS 

Most  of  the  backup  flight  instruments  in  the  cockpit  simulator  were 
inoperable  so  that  the  pilot  was  forced  to  use  the  information  displayed 
on  the  HUD  and  HSF  to  maintain  control  of  the  cockpit  simulator. 

However,  the  following  instruments,  switches  and  indicators  were  operable 
and  available  for  use  by  the  pilot  during  testing: 
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a)  Angle  of  Attack  (AOA).  The  AOA  Indicator  operated  through  a 
range  from  0  through  30  units. 

b)  HUD  Declutter  Switch.  A  three-position  switch  on  the  HUD 
control  panel  was  implemented  by  which  the  pilot  could  select  the  amount 
of  information  to  be  presented  on  the  HUD.  In  all  three  switch 
positions,  the  horizon,  ladder,  flight  director,  flight  path  marker,  and 
weapons  symbology  were  presented  on  the  HUD.  In  one  of  the  positions, 
only  the  aforementioned  symbology  was  presented.  In  a  second  position, 
exact  readouts  of  the  altitude,  airspeed,  heading  and  weapon  option  also 
appeared.  In  the  third  position,  the  HUD  presented  altitude,  airspeed, 
and  heading  scales  as  well. 

c)  Master  Arm  Switch.  Had  to  be  in  the  arm  position  in 
order  to  deliver  weapons. 

d)  Stick  Switches.  Trim  button  adjusted  stick  to  neutral 
position.  Bomb  release  button  enabled  a  selected  weapon  option  to  be 
released. 

Engine  instruments  provided  were: 

e)  RPM.  Indicated  engine  speed  in  percent  RPM.  The 
Instrument  was  calibrated  from  0  -  100%.  The  normal  operating  range  was 
52  -  100%. 

f)  Turbine  Outlet  Pressure  (TOP).  TOP  was  used  as  an 
indication  of  engine  performance.  Calibrated  In  Inches  of  mercury,  the 
operating  range  was  25  -  45  in.  Hg. 

g)  Turbine  Outlet  Temperature  (TOT).  Indicated  TOT  in 
degrees  C  (pointer  and  digital  readout).  The  usable  range  was  0  -  1000 
degrees  C. 

h)  Oil  Pressure.  Indicated  engine  oil  system  pressure  In 
psl.  The  Instrument  was  calibrated  0  -  60  psl  with  a  normal  operating 
range  of  27  -  53  psl. 
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APPENDIX  B 

SITUATION  DISPLAY  FORMATS 

As  mentioned  in  subsection  lb  of  Section  II,  a  total  of  27  formats 
were  developed.  An  example  of  one  of  the  formats  from  each  of  the  three 
density  levels  (10,  20,  and  30)  symbols  is  shown  in  Figures  Bl,  B2,  and 
B3.  Each  pilot  was  asked  one  question  about  each  format  in  the 
achromatic  shape-coded  condition,  and  one  about  it  in  the  redundant 
color-  and  shape-coded  condition.  However,  given  the  experimental 
design,  this  meant  that  he  would  see  each  format  once  in  the  morning  and 
again  in  the  afternoon  several  hours  later.  Thus,  there  was  very  little 
chance  of  his  recalling  the  format. 
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APPENDIX  C 

EXPERIMENTERS'  CONSOLE  AND  SIMULATOR  FACILITIES 
1.  EXPERIMENTERS'  CONSOLE 

The  console  was  equipped  with  CRT  displays  and  status  light 
matrices  which  provided  the  experimenters  with  the  capability  of 
monitoring  the  displays  in  the  simulator  and  the  actual  switch  actions. 

A  layout  of  the  experimenters'  console  is  shown  in  Figure  Cl.  The 
following  list  specifies  the  functions  allocated  to  each  piece  of 
equipment  on  the  console.  Each  letter  refers  to  the  notations  used  on 
the  layout. 

A  *  Experimenters'  status  display  (Figure  9);  presented  flight  and 
task  event  information 

B  3  Repeater  display  of  the  multifunction  switch  legends  on  the  MFK 

C  *  Repeater  display  of  the  SF 
D  *  Repeater  display  of  the  HUD 
E  3  A  second  repeater  display  of  the  SF 
F  *  Repeater  display  of  the  HSF 

G  3  Status  panel  lights  (Each  status  light  stayed  lit  as  long  as 
the  corresponding  switch  in  the  cockpit  was  activated.) 

H  3  Master  power  switch  for  facility 

I  3  Abort  switch  for  McFadden  flight  control  systems 

J  3  Interphone  options  (Note:  the  pilot's  mike  was  always  hot) 

K  •  On/off  switch  for  interphone  system 

L  3  Switch  enabled  communication  between  Experimenters  #1  and  #2 
M  3  Switch  enabled  Experimenters  #1  and  #2  to  communicate  with 

pilot 

N  3  Switch  enabled  communication  between  all  three  experimenters 
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0  =  Switch  enabled  communication  between  all  three  experimenters 
and  the  pilot 

P  =  Volume  control  for  Experimenter  #l's  headset 

Q  =  Voice  recorder  options 

R  =  Pause  switch  for  voice  recorder 

S  *  Run  switch  for  voice  recorder 

T  *  Reset  switch  for  McFadden  system 

U  *  Pre-event  switch;  activation  initiated  thirty  seconds  of  flight 
data  recording 

V  =  Event  marker  switch;  activation  started  recording  of  flight 
data  and  task  event  data,  started  console  timer,  and  either  unlocked  MFK 
or  displayed  new  symbol  set 

W  =  Mission  complete  switch  (guarded);  activation  initiated  the 
computerized  data  reduction  procedures 

X  =  Run  switch  for  simulation 

Y  *  Track  advance;  used  after  the  last  symbology  task  in  the  CRUISE 
segment  to  advance  to  the  CCIP  AUTO  segment 

Z  *  Keyboard  unlock  switch;  activation  unlocked  MFK  in  those  task 
events  where  recording  terminated  after  the  pilot  entered  incorrect  legal 
digits 

A1  *  Indicated  whether  tape  recording  was  continual  or  voice 
activated 

A2  *  Hold  switch  for  simulation 

A3  *  Emergency  1  switch  (Originally  Intended  as  a  means  of 
preventing  programmed  emergency  conditions  from  occurring  as  scheduled. 
Not  used  In  this  experiment.) 

A4  *  Segment  complete  switch  (guarded);  activation  terminated  test 
segment  and  automatically  updated  the  controls/displays  configuration  to 
that  specified  by  the  next  higher  matrix  number.  The  automatic 
sequencing  could  be  overridden  via  input  on  the  terminal.  (Not  used  in 
this  experiment.) 
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A5  s  Task  abort  switch  (guarded);  activation  terminated  recording  of 
task  event  data  and  initialized  system  for  next  task  event 

A6  *  Volume  control  for  Experimenter  #2  headset. 

A7  =  Hardware  configuration  switches;  the  switches  indicated  which 
set  of  function  select  switches  or  multifunction  switches  were  activated 
(front  left  panel  or  left  console)  and  which  DEK  was  activated  (left  or 
right  console)  (Not  used  in  this  experiment). 

2.  SIMULATION  FACILITIES 

The  simulator  consisted  of  interconnected  facilities  as  shown  in 
Figure  B2.  A  functional  description  of  each  system  element  is  provided 
below. 

a.  POP  11/50 

Configuration  Control  -  used  to  set  up  the  cockpit 
controls/displays  configuration  prior  to  each  flight. 

Display  Assembly  -  generated  image  lists  to  be  further 
processed  by  the  Ramtek  raster  symbol  generator.  Data  from  the 
simulation  models  was  used  for  the  HUDt  HSF,  and  SF  formats. 

Map  Driver  -  provided  output  control  of  map  data  to  the  Ramtek 
symbol  generator. 

Keyboard  Logic  -  processed  incoming  switch  data  and  determined 
the  display  state  of  all  the  keyboards. 

Flight  Control  Sampling  and  Scaling  -  buffered  and  scaled 
flight  control  data  to  be  used  by  simulation  models. 

Simulation  Models  -  provided  all  necessary  aircraft  parameters 
to  be  used  in  display  processing. 
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Data  Recording  -  recorded  cockpit  display  parameter  data  on 
magnetic  tape. 

Data  Reduction  -  an  off-line  program  reduced  the  raw  real-time 
recorded  data  into  meaningful  data  that  could  be  analyzed. 

y'* 

b.  Ramtek 

Display' Generation  -  processed  image  lists  to  display  HUD, 
HSFvand  SF. 

c.  Cockpit 

Keyboard  Input/Output  -  provided  a  switch  image  buffer  of  all 
cockpit  switch  states  to  be  sampled  by  the  PDP  11/50.  Also  decoded 
keyboard  display  data  being  set  from  the  PDP  11/50. 

Flight  Control  -  Digitized  analog  stick,  rudder,  and  thrust 
control  inputs  and  buffered  the  resultant  data  for  transmission  to  the 
PDP  11/50. 


d.  Support  Equipment 

Console  Terminal  -  system  operators  input/output  device  to  the 

PDP  11/50. 

Printer  and  Card  Reader  -  hard  copy  input/output  to  the  PDP 

11/50. 


Disk  Drive  -  mass  storage  device  for  the  operating  system. 

Magnetic  Tape  Drive  -  mass  storage  device  for  data  collection. 

Discrete  and  Analog  Input/Output  -  input/output  port  from  the 
PDP  11/50  to  all  cockpit  and  experimenter  consoles'  subsystems. 
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APPENDIX  D 
DAILY  TEST  SCHEDULE 


The  daily  test  schedule  (Table  Dl)  indicates  the  approximate 
time  and  type  of  activity  to  train,  test,  anH  debrief  one  pilot  during 
one  day  of  the  experiment.  Times  for  controls/displays  familiarization, 
training  missions,  test  missions,  data  verification,  and  debriefing  are 
indicated  in  the  schedule  provided.  As  was  mentioned  in  Section  III, 
each  pilot  participated  in  a  three  hour  briefing  at  his  home  base  prior 
to  the  on-site  testing. 


TABLE  Dl 

DAILY  TEST  SCHEDULE 


ACTIVITY* 

Cockpit  Briefing 
Break 

Training  Mission  for  Coding 
Condition  I 
Break 

Data  Mission  1 
Break 

Data  Mission  2 
Debriefing  Questionnaire 
for  Coding  Condition  I 
Lunch 

Training  Mission  for  Coding 
Condition  II 
Break 

Data  Mission  3 
Break 

Data  Mission  4 

Debriefing  Questionnaire  for 
Coding  Condition  II 
Final  Debriefing  Questionnaire 


*See  Section'*!!!  Test  Procedure. 


APPROXIMATE  TIMES 

0830  -  0930 
0930  -  0945 
0945  -  1005 

1005  -  1010 
aCIO  -  1040 
:040  -  1055 
1055  -  1125 
1125  -  1155 

1155  -  1300 
1300  -  1320 

1320  -  1325 
1325  -  1355 
1355  -  1410 
1410  -  1440 
1440  -  1510 

1510  -  1550 
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APPENDIX  E 

INITIAL  CONDITIONS  AND  COMMANDED  FLIGHTPATH 
FOR  DATA  MISSION  ONE 

This  experiment  conducted  four  training  missions  and  four  data 
missions,  initial  conditions  and  commanded  flightpath  information  for 
Data  Mission  One  are  detailed  below. 

The  cockpit  was  in  the  following  configuration  at  the 
initialization  of  each  flight: 

a)  Flight  mode  switches  -  CRUISE  flight  mode  switch  activated. 

b)  HUD  -  CRUISE  mode  symbology  appeared  (see  Figure  3).  A  true 
airspeed  (TAS)  of  350  knots,  and  an  altitude  of  1228  feet  was  shown. 

c)  HSF  -  CRUISE  mode  symbology  appeared  (see  Figure  4).  The 
aircraft  was  located  in  the  center,  groundtrack  up,  with  a  situation 
format  of  30  symbols  around  it.  No  symbology  questions  were  asked  about 
this  first  situation  format. 

d)  SF  -  CRUISE  mode  symbology,  consisting  of  communication  and 
status  information  appeared  (see  Figure  5). 

e)  MFK  -  The  MFK  Tailored  page  appeared  (see  Figure  6).  However, 
the  MFK  was  Inoperative  until  Experimenter  #1  unlocked  it  from  the 
experimenters'  console. 


INITIAL  CONDITIONS  FOR  EACH  SEGMENT  OF  DATA  MISSION  ONE 
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APPENDIX  E 

INITIAL  CONDITIONS  AND  COMMANDED  FLI6HTPATH 
FOR  DATA  MISSION  ONE 

This  experiment  conducted  four  training  missions  and  four  data 
missions,  initial  conditions  and  commanded  flightpath  information  for 
Data  Mission  One  are  detailed  below. 

The  cockpit  was  in  the  following  configuration  at  the 
initialization  of  each  flight: 

a)  Flight  mode  switches  -  CRUISE  flight  mode  switch  activated. 

b)  HUD  -  CRUISE  mode  symbology  appeared  (see  Figure  3).  A  true 
airspeed  (TAS)  of  350  knots,  and  an  altitude  of  1228  feet  was  shown. 

c)  HSF  -  CRUISE  mode  symbology  appeared  (see  Figure  4).  The 
aircraft  was  located  in  the  center,  groundtrack  up,  with  a  situation 
format  of  30  symbols  around  it.  No  symbology  questions  were  asked  about 
this  first  situation  format. 

d)  SF  -  CRUISE  mode  symbology,  consisting  of  communication  and 
status  information  appeared  (see  Figure  5). 

e)  MFK  -  The  MFK  Tailored  page  appeared  (see  Figure  6).  However, 
the  MFK  was  inoperative  until  Experimenter  #1  unlocked  it  from  the 
experimenters'  console. 


Figure  El.  Commanded  Flight  Path  for  Data  Mission  One 
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APPENDIX  F 
MISSION  ONE  SCRIPT 

Bracketed  material  £  J  is  explanatory  information  not  read 
by  Experimenter  #2.  "Dragon  10"  is  the  pilot’s  call  sign.  Note 
how  at  the  conclusion  of  each  task  (second  line  of  each  task 
narrative).  Experimenter  #2  acknowledges  the  pilot’s  symbology 
question  answer  or  his  completion  of  a  MFK  task.  The  first  two 
lines  of  the  script  are  to  establish  that  the  headsets  are 
operational . 

DRAGON  10  ,  THIS  IS  FRANKFURT  CONTROL,  SQUAWK  IDENT. 


DRAGON  10  ,  I  HAVE  RADAR  CONTACT,  CONTINUE  ON  COURSE. 

(pre-event] 

(task  1  symbology) 


DRAGON 

10  , 

DOES  QUADRANT  3  OR  QUADRANT  4  HAVE  THE 

MOST  UNKNOWN  AIRCRAFT? 

DRAGON 

10  , 

ROGER. 

[pre-event] 

[task  2  symbology] 

dragon 

10  , 

HOW  MANY  ENEMY  AIRCRAFT  ARE  HEADING 

TOWARD  9  O’CLOCK? 

dragon 

10  , 

ROGER. 

[pre-event] 

[task  3  UHF 

CHANNEL  CHG 

1 

DRAGON 

10  , 

CONTACT  ME  NOW  ON  INDIA  FREQUENCY. 

DRAGON 

10  , 

YOU'RE  LOUD  AND  CLEAR,  HOW  ME? 

DRAGON 

10  , 

ROGER. 
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[pre-event] 

[TASK  4  SYMBOLOGY] 

DRAGON  10  ,  DOES  QUADRANT  1  OR  QUADRANT  4  HAVE  THE 

MOST  ENEMY  AIRCRAFT  HEADING  TOWARD 
12  O'CLOCK? 

DRAGON  10  ,  ROGER. 

[PRE -EVENT] 

(task  5  symbology] 

DRAGON  10  ,  HOW  MANY  UNKNOWN  AAA  SITES  ARE  THERE? 

DRAGON  10  ,  ROGER. 

[PRE- EVENT] 

(TASK  6  SYMBOLOGY] 

DRAGON  10  ,  WHICH  QUADRANT  HAS  ALL  OF  THE  UNKNOWN 

AIRCRAFT  HEADING  TOWARD  12  O'CLOCK? 

DRAGON  10  ,  ROGER. 

[PRE-EVENT] 

[TASK  7  IFF  MODE  CHG,  IFF  IN/OUT] 

DRAGON  10  t  SQUAWK  MODE  3  IN  AND  SET  MODE  3  CODE  TO 

VICTOR. 

DRAGON  10  t  READ  YOUR  SQUAWK  5  BY. 

[PRE-EVENT] 

[TASK  8  SYMBOLOGY] 

DRAGON  10  ,  WHICH  QUADRANT  HAS  THE  ENEMY  AAA? 

DRAGON  10  ,  ROGER,  TRAFFIC  AT  10  O'CLOCK,  CROSSING 

RIGHT  TO  LEFT. 
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[PRE-EVENT] 

[TASK  9  IFF  MODE  CHG] 

DRAGON  10  ,  CHANGE  YOUR  MODE  3  SQUAWK  TO  XRAY. 

DRAGON  10  ,  I  READ  YOUR  SQUAWK. 

[PRE-EVENT] 

[TASK  10  TACAN  CHANNEL  CHG] 

DRAGON  10  ,  TUNE  MANSHACH  TACAN,  NOVEMBER  3. 

DRAGON  10  ,  ROGER,  ARE  YOU  ON  TOP? 

[PRE-EVENT] 

[TASK  11  SYMBOLOGY] 

DRAGON  10  ,  DOES  QUADRANT  1  OR  QUADRANT  4  HAVE  MORE 

ENEMY  AIRCRAFT  HEADING  TOWARD  6  O'CLOCK? 
DRAGON  10  ,  ROGER,  WHAT’S  YOUR  ALTITUDE? 

[PRE-EVENT] 

[TASK  12  FLY  TO  WAYPT  NO] 

DRAGON  10  ,  YOU  ARE  CLEARED  FROM  YOUR  PRESENT  POSI¬ 

TION  TO  NOVEMBER  12. 

DRAGON  10  ,  YOU  HAVE  TRAFFIC  AT  12  O'CLOCK  LOW,  FAST- 

MOVER. 

[PRE-EVENT] 

[TASK  13  SYMBOLOGY] 

DRAGON  10  ,  WHICH  QUADRANT  HAS  THE  UNKNOWN  SAMS? 

DRAGON  10  ,  ROGER,  CONTINUE. 
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[PRE-EVENT] 

[TASK  14  SYMBOLOGY] 

DRAGON 

10  , 

HOW  MANY  ENEMY  ARI CRAFT  ARE  HEADING 

DRAGON 

10  , 

TOWARD  12  O’CLOCK? 

ROGER. 

[TRACK  ADVANCE  HIT] 

[PRE-EVENT] 

[TASK  15  WPN  OPT  SELECTION] 

DRAGON  10  .  SELECT  WEAPON  OPTION  2.  CLEARED  FOR  DROP 
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0  132500E 

12  9 

16  7 

Figure  G1 .  Modified  Flight  Plan  (AF  Form  70)  for  Data  Mission  One 
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APPENDIX  H 

'  DEBRIEFING  QUESTIONNAIRES  AND  RESULTS 

Immediately  after  each  symbol  coding  condition  was  evaluated  (two 
data  flights),  the  pilot  was  given  a  questionnaire  concerning  the 
symbology  set  and  its  coding  condition.  A  final  debriefing 
questionnaire  was  administered  following  the  completion  of  all  data 
flight,  eliciting  subjective  evaluation  of  the  two  coding  conditions,  the 
symbology,*  the  MFK  logic,  and  the  overall  simulation  quality. 

Each  of  these  three  questionnaires  is  included  in  this  Appendix 
with  a  tally  of  all  of  the  pilots'  responses.  Nonparametric  Kolmogorov- 
Smirnov  tests  of  significance  (Reference  10),  were  conducted  on  this 
data.  The  results  are  reported  where  the  probability  associated  with  the 
observed  value  of  the  maximum  deviation  is  smaller  than  £  =  .05.  Also 
Included  on  each  questionnaire  is  a  composite  of  both  the  comments 
pilots  wrote  in  on  their  questionnaires  and  comments  they  made  during  the 
flights.  Occasionally,  editorial  comments  are  necessary  to  clarify  the 
comment  or  the  context  in  which  it  was  made.  These  editorial  comments 
are  contained  within  brackets. 


during  testing  and  questionnaire  administration,  the  symbology  was 
referred  to  as  threat  symbology.  After  the  study  it  was  decided  to 
change  the  terminology  to  combat  situation  symbology  since  friendly  and 
unknown  symbols  were  Included. 
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COMMENTS: 

(•)  display  at  times  was  distorted  to  look  like  ^  ,  which 
was  somewhat  similar  to  r7l . 

No  problem  with  orientation,  only  symbol  sets  for  aircraft 
and  AAA. 

-  Unknown  symbol  was  easiest  to  detect  in  any  orientation. 
The  symbols  for  friendly  and  enemy  were  easily  confused  — 
the  display  would  cause  friendly  symbol  to  often  look  like 
enemy  symbol. 

-  Once  the  (•  looked  like  [•  then  changed  to  and  back  to 

G- 

-  In  the  black  and  white  mode,  the  half-moon  symbols  and  the 
spear-shaped  symbols  were  difficult  to  distinguish. 

often  was  confused  with  V  —  visually  ^  was  often 
thought  of  as  unknown. 

Actually  has  nothing  to  do  with  the  clock  position,  merely 
the  identification  when  the  symbology  was  shifting. 

The  friendly  versus  enemy  symbols  are  quite  easily  confused. 
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2.  Did 

you  have 

any  difficulty 

in  recognizing  each 

of  the 

following  symbols  at  first  glance? 

(For  example,  did  you 

have  to 

squint. 

move  your 

head  forward. 

or  stare  at  it  for 

several 

seconds 

before  you 

could  make  it  out?) 

Never 

Very  Little 

Some  Quite  a  Bit 

Always 

S  *<1) 

12 

3 

1 

A  *(2 ) 

12 

3 

1 

1  1 *(3 ) 

7 

7 

2 

A* 

3 

3 

6  4 

1 

O  (4) 

8  6 

4 

*Some  pilots  did  not  respond. 


(1)  D=0.550,  £  <  0.01 

(2)  D=0 . 550,  £  <  0.01 

(3)  D=0.424,  £  <  0.01 

(4)  D=0.400,  £  <  0.01 
COMMENTS: 

[See  note  l]  ^  display  at  times  was  distorted  to  look  like 
/'is,  which  was  somewhat  similar  to  r¥1  . 

[A  (7s  -  Always]  Again,  same  problem.  In  black  and 

white,  the  spear-shaped  symbols  and  the  half-moon  shapes 
were  difficult  to  distinguish  at  first  glance. 

White  symbols  near  the  top  and  bottom  blended  in  with  the 
other  information  presented  and  were  difficult  to  pick  out. 
Symbols  are  too  small  and  have  poor  resolution. 

-  X  hair  on  scope  was  the  problem. 

When  adjacent  to  other  numerical  information,  especially  at 
the  limits  of  the  scope,  it  required  extra  time. 
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3.  How  much  do  you  feel  the  following  symbols  "looked  like" 
what  they  were  portraying? 


S(l) 

A(2) 


A 

r\ 


Not  at  All 


Very  Little 
1 


Somewhat 


Quite  a  Bit 
4 


Completel 

9 


(1)  D=0.322,  £  <  0.05 

(2)  0=0.322,  £  <  0.05 
COMMENTS: 


-  Very  Little]  Confused  with  A  very  much.  Computer 
generated  symbol  is  hard  to  differentiate. 

A  dot  was  too  abstract  to  represent  an  aircraft  for  me, 
particularly  a  threat. 

See  note  1  pa*  display  at  times  was  distorted  to  look  like 
T ,  which  was  somewhat  similar  to  T*  . 

Differentiation  is  poor  at  best,  even  under  optimum 
conditions  --  unacceptable  in  daylight,  under  "G"  loading, 
also  sun  glare. 

As  soon  as  you  realize  what  they  represented  identification 
was  O.K. 

S  to  me  means  Search,  therefore,  nonthreat. 

Without  color,  the  shapes  lose  a  lot  of  their  impact  value 
and  distinguishability. 

They  don’t  have  to  look  exactly  like  what  they  are 
portraying . 

Airplanes  should  be  shaped  like  "A's". 
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4.  How  would  you  rate  the  adequacy  of  aircraft  symbol  orien¬ 
tation  to  denote  aircraft  heading  (A  <$  )?(1) 

Very  Poor  Poor  Adequate  Good  Excellent 

*  _  _ 1_  _ 1_  _ 7_  8 

(1)  D=0.482,  £  <  0.01 

*One  respondent  marked  both  "good"  and  "excellent".  These 
responses  were  dropped. 

COMMENTS: 

Symbols  are  only  on  cardinal  heading. 

[Poor]  Takes  too  long  to  figure  out  heading  at  a  glance. 
Not  too  bad  when  you  know  in  advance  you '  re  looking  for 
aircraft  heading  towards  a  specific  direction. 

[Good]  Direction  was  indicated  O.K. 
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5 .  During  the  experiment  you  were  asked  to  perform  three 
different  types  of  threat  tasks:  Searching,  Counting,  and 

Comparing.  Rate  how  easy  it  was  for  you  to  perform  each  of  these 
types  of  tasks. 

Very  Moderately  Moderately  Very 

Easy  Easy  No  Opinion  Difficult  Difficult 

Searching  (In 
what  quadrant  is 

the 's' located?)  (1)  _ 3_  10  _  _ 4_  1 

Counting  (Count 

the  )  _ 1_  _ 4_  _ 1_  10  2 

Comparing  (Does 
quadrant  1  or  2 

have  more  T  ?)  _ 1_  _ 8_  _  _ 9_  _ 

(1)  D=0.322,  p  <  0.05 
COMMENTS: 

Without  color,  the  search  task  becomes  twice  as  difficult, 
which  includes  the  requirement  to  count  or  compare. 

Difficulty  was  experienced  when  comparing  friendly  shapes 
with  enemy  shapes. 

-  Major  problem  encountered:  sometimes  misinterpreted  for 

&  . 

These  tasks  cannot  be  done  at  a  glance.  It  takes  some 
serious  study  of  the  scope. 
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All  threats  had  to  be  studied  to  first  find  the  ones  looking 
for,  i.e.,  friendly,  unknown,  or  enemy,  then  those  threats 
had  to  be  studied  to  find  the  type  of  threats  questioned 
about.  This  as  opposed  to  the  color  coded  ones  took  more 
time. 

Color  makes  the  difference. 

I  really  don't  believe  your  questions  are  the  correct  ones 
for  the  data  you  are  looking  for.  Yes,  searching,  counting, 
and  comparing  are  performed — but  there  is  very  little 
"scope  interpretation".  With  the  system  of  asking  the 
question  in  advance,  you  can  key  your  mind  on  the  symbology, 
quadrant,  and  direction  of  the  different  tasks.  I  think 
questions  should  be  designed  for  scope  interpretation — 
what's  really  there,  rather  than  searching  for  specific 
threats . 
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MISCELLANEOUS  COMMENTS 

The  quadrant  dividing  line  caused  me  to  confuse  these  two 
symbols  in  one  case,  A  . 

Every  now  and  then  the  numerical  information  at  the  bottom 
of  the  display  appeared  very  close  to  the  threat  information 
causing  a  little  delayed  reaction. 


AFWAL-TR-82-3080 


COMMENTS : 

-  Didn't  look  for  A  ^  <— i  symbol.  Just  looked  for  color  then 
looked  to  see  if  SAM,  aircraft  or  AAA  by  symbol. 

With  colors,  I  found  the  symbology  to  be  fairly  straight¬ 
forward. 

The  color  differences  give  immediate  distinction  between 
enemy,  friendly,  and  unknowns.  Only  problem  was  within  a 
color  group  remembering  what  symbol  you're  looking  for. 
A  little  confusing  between  half-moon  and  spear  shapes. 

Minor  tendency  to  confuse  symbols  for  ^  ,  that  is  A  for 

"aircraft"  versus  anti-aircraft.  One  possible  alternative 
is  use  &  for  anti-aircraft  gun  which  is  normally  the  method 
that  intelligence  will  brief  it. 

None . 

No  problem. 

-  None,  the  color  eliminated  the  problem. 

Symbols  are  confusing,  i.e.,  looks  like  an  airplane.  If 
color  is  used  the  A  «— i  could  be  eliminated.  Suggest: 


Red 

Yellow 

Green 

Enemy  AC  - 

Unkn. 

AC 

Friendly  AC 

Enemy  SAM 

s 

Unkn. 

SAM 

s 

Friendly  SAM  s 

Enemy  AAA 

A 

Unkn. 

AAA 

A 

Friendly  [aaa]  a 

The  symbols 

could 

be  larger  and 

it  might 

be  good  to  make 

symbols  different 

sizes, 

i.e. 

. ,  large 

symbols — higher 

threats . 
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2.  Did  you  have  any  difficulty  in  recognizing  each  of  the 
following  symbols  at  first  glance?  (For  example,  did  you  have  to 
squint,  move  your  head  forward,  or  stare  at  it  for  several 
seconds  before  you  could  make  it  out?) 


Never 

Very  Little 

s  *<1) 

10 

5 

A  *(2 ) 

10 

5 

■-i  *(3 ) 

10 

5 

A  *(4 ) 

10 

3 

n  o) 

8 

2 

*Some  pilots  did  not  respond. 


(1)  D=0.538,  p  <  0.01 

(2)  D=0 . 538,  £>  <  0.01 

(3)  D=0 .482 ,  £  <  0.01 

(4)  D=0 .425,  £  <  0.01 

(5)  D=0 . 344,  £  <  0.05 


Some  Quite  a  Bit  Always 

_ 1_  _  _ 

_ 1_  _  _ 

_ 2_  _  _ 

_ 3__  _  _ 

7  1  _ 


COMMENTS: 

-  Friendly  symbology  could  be  de-emphasized  a  bit.  In  the 
heat  of  battle,  I  think  you  would  only  be  concerned  with  the 
friendly's  relative  position. 

-  Not  used  to  interpreting  a  •  as  an  AI  airborne  intercept 
threat.  To  me  a  dot  is  an  unknown. 

Just  don't  like  symbology — perhaps  a  circled  (g)  would  be 
better  for  the  enemy. 

! 

!  -  The  S  and  A  had  to  be  read.  I  was  unable  to  distinguish 

i 

i 

|  between  them  at  olaiK  like  I  was  able  to  with  the  dot. 

i 

i 
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3.  How  much  do  you  feel  the  following  symbols  "looked  like" 


what  they  were  portraying? 

Not  at  All 

Very  Little 

Somewhat 

Quite  a  Bit 

Completely 

S  (1) 

2 

10 

6 

A  1 

3 

2 

6 

6 

r“'  1 

3 

5 

4 

5 

A  (2)  1 

1 

2 

7 

7 

r\  l 

4 

5 

3 

5 

(1)  D=0 .489,  £  <  0. 

01 

(2)  D=0.378,  £  <  0. 

01 

COMMENTS : 

Can  confuse  A  for  A/C  until  used  to  symbology.  Symbology  OK 
once  you  are  used  to  what  they  represent. 

The  bracket  and  half  circle  were  no  problem  with  color; 
however,  I  feel  they  may  be  a  problem  in  black-white. 

Same  as  above.  [Just  don't  like  the  symbology  —  perhaps  a 
circled (E)  would  be  better  for  the  enemy.] 

[n  a  O  —  Completely  "looked  like"  what  they  were  por¬ 
traying  because  of  color  coding. 

The  problem  may  be  confusing  the  half-moon  of  the  friendly 
with  the  spear-shape  of  the  enemy. 
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4.  How  would  you  rate  the  adequacy  of  aircraft  symbol  orien¬ 
tation  to  denote  aircraft  heading  ( £  ^  ^  )?(1) 

Very  Poor  Poor  Adequate  Good  Excellent 

*  _  _  2  _ 6_  9 

(1)  D=0 .482 ,  p  <  0.01 

*One  respondent  marked  midway  between  "poor"  and  "adequate." 
This  response  was  dropped. 

COMMENTS: 

You  may  consider  making  either  the  enemy  aircraft  symbol  or 
the  friendly  aircraft  symbol  larger  (if)  when  color  is  not 
used. 

Only  cardinal  headings  are  shown. 

Showing  direction  was  good  at  a  glance. 

Would  be  difficult  at  a  glance  under  stress — especially  low 
altitude. 

[Good]  No  problem  with  90/270/360/180  headings.  Other 
headings  would  be  approximate. 
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5.  During  the  experiment  you  were  asked  to  perform  three 
different  types  of  threat  tasks:  Searching,  Counting,  and 

Comparing.  Rate  how  easy  it  was  for  you  to  perform  each  of  these 
types  of  tasks. 

Very 
Easy 

Searching  ( In 
what  quadrant  is 
the  ^  located? )  ( 1 )  10 

Counting  (Count 

the )(2)  _ 5_ 

Comparing  (Does 
quadrant  1  or  2 
have  more  V  ?)(3)  _ 7_ 

(1)  D=0 . 544,  p  <  0.01 

(2)  D=0.378,  p<  0.01 

(3)  D=0.489,  p  <  0.01 
COMMENTS : 

The  color  contrast  made  it  very  simple  to  compare  numbers  of 
symbols  in  different  quadrants. 

The  pre-questioning  keys  your  thinking--thus  shortening  the 
time.  [Moderately  Easy] 

Comparing  was  the  hardest  but  none  presented  any  real 
problem. 


Moderately  Moderately  Very 

Easy  No  Opinion  Difficult  Difficult 
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6.  Do  you  feel  the  three  colors  chosen  (red,  yellow,  and  green) 
were  distinguishable  enough?  If  no,  why?  (1) 


(1)  D=0 .444,  £  <  0.01 
COMMENTS: 

Color  considerably  enhances  the  ability  of  the  pilot  to 
distinguish  between  enemy  and  friendly.  The  only  way  to  go 
with  JTIDS .  However,  some  minor  adjustments  to  symbology 
should  be  considered. 

Red  and  green  were  excellent.  Yellow  was  a  little  difficult 
when  it  was  close  to  the  HUD  [believe  pilot  meant  digital 
readouts  on  HSf]  scales  around  the  edge. 

-  Color  worked  fine! 

-  M  Do  not  think  they  would  work  in  the  daylight,  in  the 


sun  glare.  Also  with  the  dark  visor  down  it  would  all  be 
different. 

Suggest  you  drop  O  and  A  symbols  when  using  color  since 
they  are  redundant  and  use  only  basic  symbols  such  as  A 
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7.  Do  you  feel  the  saturation  of  each  of  the  colors  was 
correct?  (For  example,  was  the  green,  green  enough?)  If  no, 
why? 

No  Yes 

Green*  ( 1 )  _  17 

Yellow*(l )  _  _ 

Red*  ( 1 )  _  16 

*Some  pilots  did  not  respond. 

(1)  D=0 . 500,  p  <  0.01 
COMMENTS : 

The  symbol  'e’1  does  not  have  the  appearance  of  an  aircraft. 
Whereas  ^  and  remind  me  of  swept  wings  of  an  aircraft. 
Again,  you  may  want  to  de-emphasize  the  friendly  symbols. 
The  green  may  be  too  green. 

Much  easier  to  distinguish  threats  with  color.  Easy  to 
eliminate  many  symbols  rapidly,  thus  focusing  on  specific 
tasked  item. 

Same  as  above.  [Do  not  think  they  would  work  in  the  day¬ 
light,  in  the  sun  glare.  Also  with  the  dark  visor  down  it 
would  all  be  different.] 

I  don't  see  the  need  to  have  both  indications  of  friendli¬ 
ness/unfriendliness.  Color  alone  seems  sufficient. 

Colors  are  very  good  and  make  the  tasks  easier;  however,  the 
symbols  need  a  lot  of  work. 
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III.  FINAL  DEBRIEFING  QUESTIONNAIRE 

The  purpose  of  this  questionnaire  is  to  gather  information 
that  will  be  helpful  in  the  development  of  multifunction  displays 
and  controls  for  advanced  fighter  cockpit  applications.  The 
questionnaire  is  divided  into  four  parts: 

PART  1.  PERSONAL  DATA  -  Information  related  to  your  background 
experience  will  be  collected. 

PART  2.  THREAT  SYMBOLOGY  -  More  information  concerning  the 
threat  symbology  presented  on  the  Horizontal  Situation 
Format  will  be  collected.  You  will  be  asked  to 
respond  as  to  how  the  symbol  coding  (shape  versus  color 
and  shape)  and  the  number  of  symbols  on  the  display 
affected  your  ability  to  identify  threat  symbols. 

PART  3.  HUD  SYMBOLOGY  -  Your  opinion  on  the  acceptability  and 
useability  of  the  symbology  programmed  on  the  simu¬ 
lator's  HUD  will  be  obtained.  (Not  included  in  this 
technical  report. ) 

PART  4.  MULTIFUNCTION  KEYBOARD  (MFK)  -  Your  thoughts  regarding 
the  use  of  a  MFK  which  integrates  many  dedicated 
control  functions  onto  a  single  panel  made  up  of  multi¬ 
function  switches  will  be  recorded.  (Not  included  in 
this  technical  report. ) 

The  extent  to  which  this  questionnaire  can  contribute  to  our 
data  analysis  will  depend  largely  upon  your  candid  opinion.  Most 
of  the  questions  can  be  answered  with  a  check  (  V ) ,  but  you  are 
encouraged  to  make  further  comments.  Please  be  as  specific  as 
possible. 
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PART  1 

PERSONAL  DATA 


Name  _  Age  Mean  -  34  Years 

Organization  9  pilots;  121  Tactical  Fighter  Wing,  Rickenbacker 

AFB 

9  pilots;  178  Tactical  Fighter  Group,  Springfield 
Municipal  Airport 


Present  Assignment 


Total  Flying  Time  Mean  =  2305  Hours 


Total  Jet  Time  Mean  =  2064  Hours 


Current  Aircraft  _ A-7D 

Other  Fighter  Aircraft  Flown 

F-100 

f=3S 

A-3 

Fh55 

F-9 

FIqS 


Time  in  Type  Mean  =  493  Hours 
Time  in  Type 

13  pilots;  12807  hrs 
6  pilots:  1914  hrs 
3  pilots:  12S>0  hrs 
2  pilots;  1202  hrs 
2  pilots:  293  hrs 

2  pilots;  190  hrs 
1  pilot:  170  hrs 

1  pilot:  ~80  hrs 


Total  Years  Rated  Mean  =  10  Years 


What  types  of  Head-Up  displays  have  you  flown? 


(18)  A- 7  (Elliott) 

(  1)  F-lll  (Norden) 

(  )  F-15  (McDonnell -Douglas) 

(  1)  F-lll  Simulator 
(  1)  A-10  Simulator 
(  1)  F-15  Simulator 


(  )  In  Combat 
(  )  In  Combat 
(  )  In  Combat 
(  )  In  Combat 
(  )  In  Combat 
(  )  In  Combat 


Civilian  Job 


10  Pilots 


None 


1  Pilot 
1  Pilot 
1  Pilot 
1  Pilot 
1  Pilot 
1  Pilot 
1  Pilot 
1  Pilot 


Corporate  Pilot 

Building  Contractor 

Northrop  Marketer 

Psychotherapist 

Engineering  Psychologist 

Stock  Broker  -  Account  Executive 

Corporate  Manager 

Engineer 
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PART  2 

COMBAT  SITUATION  SYMBOLOGY 

1.  During  the  course  of  the  experiment,  you  were  asked 
questions  about  threat  displays  which  contained  either  10,  20,  or 
30  symbols.  Sometimes  the  symbols  were  all  white,  sometimes  the 
symbols  were  color-coded.  How  do  you  think  the  addition  of 
color-coding  affected  the  time  it  took  for  you  to  answer  the 
question? 

Decreased  Increased 

It  A  Great  Decreased  Did  Not  Increased  It  A  Great 
Deal  It  Some  Affect  It  It  Some  Deal 

With  10 

symbols  ( 1 )  10  7  1  _  _ 

With  20 

symbols(2 )  15  3  _  _  _ 

With  30 

symbols  (3)  16  2  _  _  _ 

(1)  D=0 . 544 ,  £  <  0.01 

(2)  D=0.633,  £  <  0.01 

(3)  D=0 . 689 ,  £  <  0.01 
COMMENTS: 

-  Color  adds  to  the  impact  value  and  distinguishability  of  the 
symbology. 

-  Felt  very  comfortable  with  color.  Color  reduced  the  time  to 
respond  considerably. 
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The  more  symbols  the  more  color  coding  helped. 

The  colors  make  the  tasks  easier,  especially  if  you  must 
look  quickly. 

Had  to  look  at  every  symbol  to  determine  what  it  was  in 
black/white,  but  with  color  only  looked  at  appropriate 
color. 

Removed  deciphering  problem  between  friendly/enemy  aircraft. 
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2.  How  do  you  think  the  addition  of  color-coding  affected  the 
number  of  errors  you  made  in  recognizing  the  threats? 

Decreased  Increased 

It  A  Great  Decreased  Did  Not  Increased  It  A  Great 
Deal  It  Some  Affect  It  It  Some  Deal 

With  10 

symbols  (1 )  7  8  3  _  _ 

With  20 

symbols  (2 )  12  6  _  _  _ 

With  30 

symbols(3 )  14  4  _  _  _ 

(1)  D=0.433,  2  <  0.01 

(2)  0=0.600,  £  <  0.01 

(3)  D=0 . 600,  2  <  0*01 
COMMENTS: 

The  more  symbols  displayed,  the  more  the  color  helped.  I 
really  liked  the  color. 

Color  symbology  eased  assigned  tasking,  e.g.,  elimination  of 
many  threats  that  on  black-white  had  to  be  integrated  adding 
to  length  of  time  required  to  complete  questions. 

When  there  is  a  doubt  between  A  and  the  color  took  the 
doubt  out  of  it.  Color  is  by  far  superior  for  threat  recog¬ 
nition. 

-  Need  color  for  quick  recognition  or  evaluation  by  a  pilot 
in  a  tense,  confused  environment,  such  as  combat. 
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Errors  could  still  be  made  because  the  symbols  are  slightly 
confusing. 

I  may  be  wrong  but  I  felt  more  comfortable  with  my  answers 
using  the  color  coding. 
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3.  If  this  simulation  had  been  an  actual  combat  mission,  how  do 
you  feel  that  the  use  of  the  threat  display  would  have  affected  a 
successful  completion  of  the  mission? 

Hindered  a  Hindered  Would  Not  Have  Helped  Helped  a 
Great  Deal  Some  Affected  It  Some  Great  Deal 

With 
color 
coded 

symbols*(l)  _  _  _  3  14 

With 
white 

symbols  1  3  3  10  _ 1 

*One  pilot  did  not  respond. 

(1)  D=0.578,  j>  <  0.01 
COMMENTS: 

Could  not  use  the  information  on  a  low  level  ingress  (below 
150'  AGL),  oral  information  is  more  valuable  when  lookout 
doctrine  is  essential.  There  simply  isn’t  enough  time  to 
interpret  the  symbology  at  low  altitude,  high  airspeed 
ingress . 

-  Threat  information  is  of  extreme  importance  on  a  combat 
mission,  in  order  for  the  pilot  to  alter  his  ingress/egress 
route,  altitude,  formation,  target  tactics,  etc.  The  more 
accurate  and  up  to  date  the  threat  information  is,  the 
better  chance  of  mission  success. 


AFWAL-TR-82-3080 


With  black/white  there  would  have  been  more  of  a  tendency  to 
overreact  to  a  friendly  aircraft  (i.e.,  breaking  into  it) 
closing  on  me. 

The  color  would  be  much  more  superior  in  obtaining  the  big 
picture  of  what  is  going  on  around  you  at  a  time  this  is 
very  critical.  This  is  especially  true  when  you  have  many 
symbols  on  the  tube. 

Using  black  and  white  with  ten  helped  some.  More  than  ten 
would  hinder.  I  feel  that  20  perhaps  30  items,  with  prac¬ 
tice,  on  color  presentation  can  be  successfully  achieved. 
Without  color  you  could  spend  too  much  time  in  the  cockpit 
whereas  the  color  lets  you  get  a  feel  of  the  situation  at  a 
glance. 

Without  the  color,  it  takes  a  longer  time  to  evaluate  what 
your  screen  is  telling  you.  This  means  less  time  for  look¬ 
ing  outside,  navigating,  and  in  general,  situation  aware¬ 
ness. 

I  probably  would  have  begun  ignoring  the  all  white  symbology 
simply  because  I  would  have  been  using  valuable  time  search¬ 
ing  through  the  clutter.  Too  much  time  was  wasted  telling 
"good"  from  "bad"  with  black/white.  Recommend  option  to 
deselect  "good  guys"  when  using  black/white. 

The  threat  display  is  very  good  and  would  allow  you  to  avoid 
the  threats. 
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MISCELLANEOUS  COMMENTS 
Color  is  the  better  of  the  two,  by  far. 

Do  we  need  direction  on  &  or  other  than  aircraft.  Could 
eliminate  some  display  on  A  or  S. 


r». 


The  majority  of  the  pilots  commented  that  they  liked  the 
color/shape  coding  condition  better  than  the  shape  (monochrome) 
condition.  With  color,  the  pilots  could  identify  threat  states 
more  easily. 

Most  of  the  pilots  further  commented  that  they  had  diffi¬ 
culty  differentiating  between  enemy  aircraft  and  friendly  air¬ 
craft  due  to  the  symbol  distortion  when  they  shifted  location. 
Some  pilots  suggested  making  the  symbol  more  of  a  delta  or  adding 
longer  wings. 

This  symbol  confusion  was  true  also  for  the  ground  threats. 
There  were  instances  where  and  ^  were  confused  with  A  and 

A  few  pilots  mentioned  that  the  symbol  for  AAA,  A,  meant 
aircraft  to  them  at  times.  In  fact,  some  errors  on  threat  tasks 
were  due  to  the  pilot  looking  at  the  AAA  symbol  rather  than  the 
airplane  symbol. 

The  use  of  the  hats  on  the  color-coded  ground  threat  symbols 
was  thought  to  be  unnecessary  by  a  few  pilots.  One  pilot  felt 
that  eliminating  the  hats  would  decrease  recognition  time. 

The  quadrant  numbering  system  was  confusing  to  several 
pilots.  Two  pilots  commented  that  they  would  prefer  the  qua¬ 
drants  be  numbered  counter-clockwise  starting  in  the  upper-right 
quadrant  and  another  two  would  prefer  they  be  numbered  clockwise 
starting  in  the  upper-left  quadrant.  Another  pilot  mentioned  he 
would  prefer  the  .quadrants  numbered  as  follows:  qtti 


AFWAL-TR-82-3080 


It  was  also  noted  that  more  of  a  delineation  between  the  quadrant 
numbers  and  the  readouts  on  the  HSD  would  help. 

OTHER  COMMENTS: 

-  Threat  symbol  size  too  small. 

-  Enemy  aircraft  threats  traveling  3  and  9  o' clocks  are 
easier  to  identify  than  enemy  aircraft  threats  traveling 
12  and  6  o' clocks. 

-  Threat  symbology  used  on  A-7  RHAW  scope  may  be  better  (U 
for  Unknown  aircraft  symbol,  etc.). 
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COCKPIT  DISPLAYS 

COMMENTS: 

-  Like  having  present  position  readout  on  MPD  where  it  can 
be  seen  easily. 

-  In  HSD,  readout  indicating  number  of  minutes  to  the  next 
waypoint,  the  readout  should  be  in  seconds  rather  than 
tenths  of  minutes  when  the  aircraft  is  less  than  a  minute 
away  from  the  waypoint. 

Many  pilots  commented  that  the  slashes  through  the  zeros 
made  them  difficult  to  differentiate  from  6s  and  8s. 
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SIMULATOR  FACILITY 
COMMENTS: 

-  Trouble  seeing  kneeboard. 

-  Stick  feels  heavy. 

-  RPM  gauge  does  not  have  a  fine  scale.  Makes  it  difficult 
to  fine  tune  throttle. 

-  Stick  seems  too  far  forward. 
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